USB mini-summit at LinuxCon Vancouver

USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.

What:	USB mini-summit
When:	Tuesday, August 16th, all day
Where:	At the conference venue, room TBD pending confirmation from
	Angela Brown.

Proposed topics include USB virtualization, and improved bandwidth APIs
between the USB core and drivers (especially webcam drivers).  See the
detailed topic list below.  Anyone is also welcome to propose or show up
with a USB related topic.  MUSB?  USB 3.0 gadget drivers?  USB-IP?

The USB mini-summit does overlap with the virtualization mini-summit by
a day, but I'm hoping we can schedule talks so some of the
virtualization folks can make it to the USB mini-summit.  The other
option was on Friday during the conference which was not ideal.

Proposed topics:

Topic 1
-------

The KVM folks suggested that it would be good to get USB and
virtualization developers together to talk about how to virtualize the
xHCI host controller.  The xHCI spec architect worked closely with
VMWare to get some extra goodies in the spec to help virtualization, and
I'd like to see the other virtualization developers take advantage of
that.  I'd also like us to hash out any issues they have been finding in
the USB core or xHCI driver during the virtualization effort.


Topic 2
-------

I'd also like to get the V4L and audio developers who work with USB
devices together with the core USB folks to talk about bandwidth
management under xHCI.

One of the issues is that since the xHCI hardware does bandwidth
management, not the xHCI driver, a schedule that will take too much
bandwidth will get rejected much sooner than any USB driver currently
expects (during a call to usb_set_interface).  This poses issues, since
most USB video drivers negotiate the video size and frame rate after
they call usb_set_interface, so they don't know whether they can fall
back to a less bandwidth-intensive setting.  Currently, they just submit
URBs with less and less bandwidth until one interval setting gets
accepted that won't work under xHCI.

A second issue is that that some drivers need less bandwidth than the
device advertises, and the xHCI driver currently uses whatever periodic
interval the device advertises in its descriptors.  This is not what the
video/audio driver wants, especially in the case of buggy high speed
devices that advertise the interval in frames, not microframes.  There
needs to be some way for the drivers to communicate their bandwidth
needs to the USB core.  We've known about this issue for a while, and I
think it's time to get everyone in the same room and hash out an API.

(I will send out an API proposal later this month.)

Hope to see you there!

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> Topic 1
> -------
> 
> The KVM folks suggested that it would be good to get USB and
> virtualization developers together to talk about how to virtualize the
> xHCI host controller.  The xHCI spec architect worked closely with
> VMWare to get some extra goodies in the spec to help virtualization, and
> I'd like to see the other virtualization developers take advantage of
> that.  I'd also like us to hash out any issues they have been finding in
> the USB core or xHCI driver during the virtualization effort.

Do people really want to virtualize the whole xHCI controller, or just
specific ports or devices to the guest operating system?

If just specific ports, would something like usbip be better for virtual
machines, with the USB traffic going over the network connection between
the guest/host?

> Hope to see you there!

Thanks for putting this together.

Do we need to sign up somewhere to give the organizers a sense of how
many people will be attending?

thanks,

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Gerd Hoffmann]

   Hi,

>> The KVM folks suggested that it would be good to get USB and
>> virtualization developers together to talk about how to virtualize the
>> xHCI host controller.  The xHCI spec architect worked closely with
>> VMWare to get some extra goodies in the spec to help virtualization, and
>> I'd like to see the other virtualization developers take advantage of
>> that.  I'd also like us to hash out any issues they have been finding in
>> the USB core or xHCI driver during the virtualization effort.
>
> Do people really want to virtualize the whole xHCI controller, or just
> specific ports or devices to the guest operating system?

SR/IOV support is an optional xHCI feature.  As I understand it you can 
create a VF which looks like a real xHCI controller.  This is partly 
done in hardware and partly by software.  Then you can assign it some 
ressources (specific ports) and pass it to the guest.

> If just specific ports, would something like usbip be better for virtual
> machines, with the USB traffic going over the network connection between
> the guest/host?

There are several ways depending on the use case.  Usually the guest 
sees a (fully software emulated) host adapter with usb devices 
connected, where the usb devices can be (a) emulated too or (b) real usb 
devices passed through to the guest.  The later is done by passing the 
guests requests to the real device via usbfs.

One problem with emulating usb fully in software is the polling design 
of the hardware which makes the emulation quite cpu intensive.  Using a 
xHCI VF should help here alot, but works for the pass through use case 
only of course.

cheers,
   Gerd

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 06/10/2011 02:21 AM, Sarah Sharp wrote:
> I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.
>
> What:	USB mini-summit
> When:	Tuesday, August 16th, all day
> Where:	At the conference venue, room TBD pending confirmation from
> 	Angela Brown.
>
> Proposed topics include USB virtualization, and improved bandwidth APIs
> between the USB core and drivers (especially webcam drivers).  See the
> detailed topic list below.  Anyone is also welcome to propose or show up
> with a USB related topic.  MUSB?  USB 3.0 gadget drivers?  USB-IP?
>

I would like to give a short presentation on / demo off the usb redirection
over the network (USB-IP replacement with a less dumb protocol) I've been
working on, followed by a questions and answers session.

For those who want to give this a try now, see:
http://hansdegoede.livejournal.com/9682.html

Sheets from the presentation I gave on this at FOSDEM:
http://people.fedoraproject.org/~jwrdegoede/usb-redir.pdf

> The USB mini-summit does overlap with the virtualization mini-summit by
> a day, but I'm hoping we can schedule talks so some of the
> virtualization folks can make it to the USB mini-summit.  The other
> option was on Friday during the conference which was not ideal.
>
> Proposed topics:
>
> Topic 1
> -------
>
> The KVM folks suggested that it would be good to get USB and
> virtualization developers together to talk about how to virtualize the
> xHCI host controller.  The xHCI spec architect worked closely with
> VMWare to get some extra goodies in the spec to help virtualization, and
> I'd like to see the other virtualization developers take advantage of
> that.  I'd also like us to hash out any issues they have been finding in
> the USB core or xHCI driver during the virtualization effort.

I'm not really happy with how the management / hand over of userspace
to kernel space drivers and vice versa works. This is a problem when
doing usb-redirection to a virtual machine. I think this is best discussed
in a separate thread, and then if needed further discussed during the
mini summit. I'll send a mail on this shortly. I'll use the same address
list as this mail, except that I'm going to cut linux-kernel from the CC list.


> Topic 2
> -------
>
> I'd also like to get the V4L and audio developers who work with USB
> devices together with the core USB folks to talk about bandwidth
> management under xHCI.
>
> One of the issues is that since the xHCI hardware does bandwidth
> management, not the xHCI driver, a schedule that will take too much
> bandwidth will get rejected much sooner than any USB driver currently
> expects (during a call to usb_set_interface).  This poses issues, since
> most USB video drivers negotiate the video size and frame rate after
> they call usb_set_interface, so they don't know whether they can fall
> back to a less bandwidth-intensive setting.  Currently, they just submit
> URBs with less and less bandwidth until one interval setting gets
> accepted that won't work under xHCI.
>
> A second issue is that that some drivers need less bandwidth than the
> device advertises, and the xHCI driver currently uses whatever periodic
> interval the device advertises in its descriptors.  This is not what the
> video/audio driver wants, especially in the case of buggy high speed
> devices that advertise the interval in frames, not microframes.  There
> needs to be some way for the drivers to communicate their bandwidth
> needs to the USB core.  We've known about this issue for a while, and I
> think it's time to get everyone in the same room and hash out an API.

Interesting, being able to tell the core how much bandwidth a device
will actually use (versus what it claims as maxpacketsize in its
descriptors) is something which we really need. I know of at least
2 usb1 webcam chipsets (and drivers) which have only 1 altsetting which
claim 1023 bytes maxpacketsize. But they also have a register which
allows the driver to configure how large the largest (iso) packet it
sends will actually be.

Currently these drivers have this "beauty" to be able to tell the
linux usb core that they won't be using 1023 as maxpacketsize but
something else, and thus function without needing full usb1 bandwidth:

struct usb_host_interface *alt;
alt = &gspca_dev->dev->config->intf_cache[0]->altsetting[1];
alt->endpoint[0].desc.wMaxPacketSize = cpu_to_le16(packet_size);
ret = usb_set_interface(gspca_dev->dev, gspca_dev->iface, 1);

Yes they are overwriting the kernels cached descriptors ...

###

Something which I would also like to bring to everyone's attention
is that we really need to fix the ehci schedule code wrt scheduling
usb1 transfers over usb2 hubs.

The current code is very broken when it comes to periodic transfers,
it basically disallows using the last microframe (let alone the
crossing of the frame boundary and using the first microframe of
the next frame).

This means that trying to submit isoc transfers with a size of 1023
will just plain fail, even if this is the only device on the entire
bus. This is one of the reasons I ended up doing the hack above, so
that these devices will at least work (be it at a decreased framerate)
through a usb2 hub.

Things become even messier when the device has a build in usb audio
microphone, often this will just not work.

Finally fixing this has become more important now then ever since
sandybridge machines (and maybe generation one core i# machines too,
I don't know) no longer have a companion controller, but instead
have an integrated usb2 hub.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> > Topic 1
> > -------
> > 
> > The KVM folks suggested that it would be good to get USB and
> > virtualization developers together to talk about how to virtualize the
> > xHCI host controller.  The xHCI spec architect worked closely with
> > VMWare to get some extra goodies in the spec to help virtualization, and
> > I'd like to see the other virtualization developers take advantage of
> > that.  I'd also like us to hash out any issues they have been finding in
> > the USB core or xHCI driver during the virtualization effort.
> 
> Do people really want to virtualize the whole xHCI controller, or just
> specific ports or devices to the guest operating system?

A host OS could chose to virtualize the whole xHCI controller if it
wanted to.  That's part of the reason why xHCI does all the bandwidth
checking in hardware, not in software.

> If just specific ports, would something like usbip be better for virtual
> machines, with the USB traffic going over the network connection between
> the guest/host?

It could be done that way too.  But that doesn't help if you're trying
to run Windows under Linux, right?  Only if all the guest OSes use the
same USB IP protocol then it would work.

> > Hope to see you there!
> 
> Thanks for putting this together.
> 
> Do we need to sign up somewhere to give the organizers a sense of how
> many people will be attending?

I private ack by email would be great.  Or you can ack by facebook:
https://www.facebook.com/event.php?eid=229532200405657  I could add the
event in upcoming.yahoo.com if anyone uses that.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
> > On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> > > Topic 1
> > > -------
> > > 
> > > The KVM folks suggested that it would be good to get USB and
> > > virtualization developers together to talk about how to virtualize the
> > > xHCI host controller.  The xHCI spec architect worked closely with
> > > VMWare to get some extra goodies in the spec to help virtualization, and
> > > I'd like to see the other virtualization developers take advantage of
> > > that.  I'd also like us to hash out any issues they have been finding in
> > > the USB core or xHCI driver during the virtualization effort.
> > 
> > Do people really want to virtualize the whole xHCI controller, or just
> > specific ports or devices to the guest operating system?
> 
> A host OS could chose to virtualize the whole xHCI controller if it
> wanted to.  That's part of the reason why xHCI does all the bandwidth
> checking in hardware, not in software.

And here I thought it did that so it would be "correct" :)

> 
> > If just specific ports, would something like usbip be better for virtual
> > machines, with the USB traffic going over the network connection between
> > the guest/host?
> 
> It could be done that way too.  But that doesn't help if you're trying
> to run Windows under Linux, right?  Only if all the guest OSes use the
> same USB IP protocol then it would work.

usbip works on Windows as well as Linux.

But how could you run Windows with a xHCI controller in a guest, as
Windows has no xHCI driver?  What would it expect to see?

> > > Hope to see you there!
> > 
> > Thanks for putting this together.
> > 
> > Do we need to sign up somewhere to give the organizers a sense of how
> > many people will be attending?
> 
> I private ack by email would be great.  Or you can ack by facebook:
> https://www.facebook.com/event.php?eid=229532200405657  I could add the
> event in upcoming.yahoo.com if anyone uses that.

Nice try, but I'm still not going to join facebook...

Re: USB mini-summit at LinuxCon Vancouver

[Alexander Graf]

Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:

> On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
>> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
>>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
>>>> Topic 1
>>>> -------
>>>> 
>>>> The KVM folks suggested that it would be good to get USB and
>>>> virtualization developers together to talk about how to virtualize the
>>>> xHCI host controller.  The xHCI spec architect worked closely with
>>>> VMWare to get some extra goodies in the spec to help virtualization, and
>>>> I'd like to see the other virtualization developers take advantage of
>>>> that.  I'd also like us to hash out any issues they have been finding in
>>>> the USB core or xHCI driver during the virtualization effort.
>>> 
>>> Do people really want to virtualize the whole xHCI controller, or just
>>> specific ports or devices to the guest operating system?
>> 
>> A host OS could chose to virtualize the whole xHCI controller if it
>> wanted to.  That's part of the reason why xHCI does all the bandwidth
>> checking in hardware, not in software.
> 
> And here I thought it did that so it would be "correct" :)
> 
>> 
>>> If just specific ports, would something like usbip be better for virtual
>>> machines, with the USB traffic going over the network connection between
>>> the guest/host?
>> 
>> It could be done that way too.  But that doesn't help if you're trying
>> to run Windows under Linux, right?  Only if all the guest OSes use the
>> same USB IP protocol then it would work.
> 
> usbip works on Windows as well as Linux.

Do you have a reliable, working usbip solution at hand that work on Windows and Linux and doesn't require real network access, which can be a no-go for some scenarios?

> 
> But how could you run Windows with a xHCI controller in a guest, as
> Windows has no xHCI driver?  What would it expect to see?

There are drivers for xhci adapters on Windows. Also, this whole discussion is pretty much future oriented - which most likely means built-in xhci drivers anywhere.

No, in all seriousity, the main reason to go for FV vs PV is that so far, the best tested and working drivers are built for real hardware. I'm still unsatisfied with the PV driver situation for virtio on Windows. It's just incredibly hard to get Windows drivers right - and open source developers sure are not good at it :)

But sure, let's talk about it during LinuxCon as well - there's a good chance you know more there than me :)

> 

Alex

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Mon, Jun 13, 2011 at 12:44:57PM +0200, Alexander Graf wrote:
> 
> Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:
> 
> > On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
> >> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
> >>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> >>>> Topic 1
> >>>> -------
> >>>> 
> >>>> The KVM folks suggested that it would be good to get USB and
> >>>> virtualization developers together to talk about how to virtualize the
> >>>> xHCI host controller.  The xHCI spec architect worked closely with
> >>>> VMWare to get some extra goodies in the spec to help virtualization, and
> >>>> I'd like to see the other virtualization developers take advantage of
> >>>> that.  I'd also like us to hash out any issues they have been finding in
> >>>> the USB core or xHCI driver during the virtualization effort.
> >>> 
> >>> Do people really want to virtualize the whole xHCI controller, or just
> >>> specific ports or devices to the guest operating system?
> >> 
> >> A host OS could chose to virtualize the whole xHCI controller if it
> >> wanted to.  That's part of the reason why xHCI does all the bandwidth
> >> checking in hardware, not in software.
> > 
> > And here I thought it did that so it would be "correct" :)
> > 
> >> 
> >>> If just specific ports, would something like usbip be better for virtual
> >>> machines, with the USB traffic going over the network connection between
> >>> the guest/host?
> >> 
> >> It could be done that way too.  But that doesn't help if you're trying
> >> to run Windows under Linux, right?  Only if all the guest OSes use the
> >> same USB IP protocol then it would work.
> > 
> > usbip works on Windows as well as Linux.
> 
> Do you have a reliable, working usbip solution at hand that work on
> Windows and Linux and doesn't require real network access, which can
> be a no-go for some scenarios?

What do you mean by "real network access"?  A virtual network connection
to the guest and host should be sufficient, right?

> > But how could you run Windows with a xHCI controller in a guest, as
> > Windows has no xHCI driver?  What would it expect to see?
> 
> There are drivers for xhci adapters on Windows.

Not well-working ones, see the windows driver mailing list as well as
the libusb mailing list for problems that people are having due to there
not being any "official" Microsoft xhci drivers shipping yet.

> Also, this whole discussion is pretty much future oriented - which
> most likely means built-in xhci drivers anywhere.

Agreed.

thanks,

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Alexander Graf]

On 13.06.2011, at 18:29, Greg KH wrote:

> On Mon, Jun 13, 2011 at 12:44:57PM +0200, Alexander Graf wrote:
>> 
>> Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:
>> 
>>> On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
>>>> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
>>>>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
>>>>>> Topic 1
>>>>>> -------
>>>>>> 
>>>>>> The KVM folks suggested that it would be good to get USB and
>>>>>> virtualization developers together to talk about how to virtualize the
>>>>>> xHCI host controller.  The xHCI spec architect worked closely with
>>>>>> VMWare to get some extra goodies in the spec to help virtualization, and
>>>>>> I'd like to see the other virtualization developers take advantage of
>>>>>> that.  I'd also like us to hash out any issues they have been finding in
>>>>>> the USB core or xHCI driver during the virtualization effort.
>>>>> 
>>>>> Do people really want to virtualize the whole xHCI controller, or just
>>>>> specific ports or devices to the guest operating system?
>>>> 
>>>> A host OS could chose to virtualize the whole xHCI controller if it
>>>> wanted to.  That's part of the reason why xHCI does all the bandwidth
>>>> checking in hardware, not in software.
>>> 
>>> And here I thought it did that so it would be "correct" :)
>>> 
>>>> 
>>>>> If just specific ports, would something like usbip be better for virtual
>>>>> machines, with the USB traffic going over the network connection between
>>>>> the guest/host?
>>>> 
>>>> It could be done that way too.  But that doesn't help if you're trying
>>>> to run Windows under Linux, right?  Only if all the guest OSes use the
>>>> same USB IP protocol then it would work.
>>> 
>>> usbip works on Windows as well as Linux.
>> 
>> Do you have a reliable, working usbip solution at hand that work on
>> Windows and Linux and doesn't require real network access, which can
>> be a no-go for some scenarios?
> 
> What do you mean by "real network access"?  A virtual network connection
> to the guest and host should be sufficient, right?

Yes, which would either mean we inject a second network adapter into the guest which needs to be configured to an IP address, hence possibly colliding with some other IP range that's in use outside, or reuse some real network connection, which is neither always given, nor always desired. We have a full new PV serial protocol (virtio-serial) just for exactly that reason: To be able to run a daemon inside the guest that does not require IP access to talk to the host.

USB/IP really has its own huge set of issues. Sure, it might be good for certain use cases, but it's definitely not a one-size-fits-all solution. But so far, doing FV emulation right has worked out a lot better for most use cases than inventing PV solutions. Real hardware simply seems to be supported by others, while we can't afford to write drivers for all OSs :). So IMHO, PV should be the last resort, when emulating real hardware utterly fails. In this case, I believe the hardware designers finally got the spec usable for us, so it should be good! :)

> 
>>> But how could you run Windows with a xHCI controller in a guest, as
>>> Windows has no xHCI driver?  What would it expect to see?
>> 
>> There are drivers for xhci adapters on Windows.
> 
> Not well-working ones, see the windows driver mailing list as well as
> the libusb mailing list for problems that people are having due to there
> not being any "official" Microsoft xhci drivers shipping yet.

Yes, which is an issue that's hopefully gone when we're finally having xHCI actually properly emulated and all the infrastructure for it laid out. We still have UHCI and OHCI emulation today already to fall back to if really necessary, but installing an xHCI driver inside the guest really isn't more work than installing USB/IP inside the guest.


Alex

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Hi Sarah/Greg,

Em 09-06-2011 21:21, Sarah Sharp escreveu:
> I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.
> 
> What:	USB mini-summit
> When:	Tuesday, August 16th, all day
> Where:	At the conference venue, room TBD pending confirmation from
> 	Angela Brown.
> 
> Proposed topics include USB virtualization, and improved bandwidth APIs
> between the USB core and drivers (especially webcam drivers).  See the
> detailed topic list below.  Anyone is also welcome to propose or show up
> with a USB related topic.  MUSB?  USB 3.0 gadget drivers?  USB-IP?
> 
> The USB mini-summit does overlap with the virtualization mini-summit by
> a day, but I'm hoping we can schedule talks so some of the
> virtualization folks can make it to the USB mini-summit.  The other
> option was on Friday during the conference which was not ideal.
> 
> Proposed topics:
> 
> Topic 1
> -------
> 
> The KVM folks suggested that it would be good to get USB and
> virtualization developers together to talk about how to virtualize the
> xHCI host controller.  The xHCI spec architect worked closely with
> VMWare to get some extra goodies in the spec to help virtualization, and
> I'd like to see the other virtualization developers take advantage of
> that.  I'd also like us to hash out any issues they have been finding in
> the USB core or xHCI driver during the virtualization effort.
> 
> 
> Topic 2
> -------
> 
> I'd also like to get the V4L and audio developers who work with USB
> devices together with the core USB folks to talk about bandwidth
> management under xHCI.
> 
> One of the issues is that since the xHCI hardware does bandwidth
> management, not the xHCI driver, a schedule that will take too much
> bandwidth will get rejected much sooner than any USB driver currently
> expects (during a call to usb_set_interface).  This poses issues, since
> most USB video drivers negotiate the video size and frame rate after
> they call usb_set_interface, so they don't know whether they can fall
> back to a less bandwidth-intensive setting.  Currently, they just submit
> URBs with less and less bandwidth until one interval setting gets
> accepted that won't work under xHCI.
> 
> A second issue is that that some drivers need less bandwidth than the
> device advertises, and the xHCI driver currently uses whatever periodic
> interval the device advertises in its descriptors.  This is not what the
> video/audio driver wants, especially in the case of buggy high speed
> devices that advertise the interval in frames, not microframes.  There
> needs to be some way for the drivers to communicate their bandwidth
> needs to the USB core.  We've known about this issue for a while, and I
> think it's time to get everyone in the same room and hash out an API.
> 
> (I will send out an API proposal later this month.)

While discussing the topics for the media workshop that will happen together
with this year's KS/2011, one issue related to the USB stack came on our
discussions: it is related to multi-function USB devices, as you can
see on this thread:
	http://www.spinics.net/lists/linux-media/msg36195.html

We have several cases of multi-function devices at the media subsystem.
For example, most TV grabber devices provides Remote Controller, Video,
audio and MPEG streaming capabilities, either implementing the standard
USB API's, or the vendor class API's. There are even some devices that
support USB storage, 3G modem and Digital TV.

Some of those devices have some resources that are mutually exclusive.

For example, some Digital Cameras can provide either access to the stored
images, or can be used as webcams. Using them as webcams automatically
deletes all stored images from it. So, a properly implemented driver
should be returning -EBUSY (or -EPERM?) if someone tries to stream for
such devices if is there any pictures stored on it. Symmetrically,
accessing the stored pictures there should return -EBUSY if the device
is streaming.

Currently, the data access for Digital Cameras is implemented via libusb,
libgphoto and gvfs, while the streaming interface is implemented via
a gspca Kernel driver.

I know that this problem were somewhat solved for 3G modems, with the usage
of the userspace problem usb_modeswitch, and with some quirks for the USB
storage driver, but I'm not sure if such tricks will scale forever, as more
functions are seen on some USB devices.

So, if we have some time, maybe we could start some discussions about that
during the USB mini-summit.

It should be noticed that the media subsystem has currently similar problems 
on devices that provide both analog and digital TV, as some resources, like 
the tuner can't be used simultaneously by the two API's. This happens even
with PCI devices. So, maybe, in the end, we'll be writing some kernel library
for resource locking, but it would be good if we can have preliminary
discussions there, and let the final discussions to happen during KS/2011. 

What do you think?

Thanks!
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> I know that this problem were somewhat solved for 3G modems, with the usage
> of the userspace problem usb_modeswitch, and with some quirks for the USB
> storage driver, but I'm not sure if such tricks will scale forever, as more
> functions are seen on some USB devices.

Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitch does.  We made that decision a while ago, and it is
working out very well.  I see no reason why you can't do it in userspace
as well as that is the easiest place to control this type of thing.

I thought we had a long discussion about this topic a while ago and came
to this very conclusion.  Or am I mistaken?

thanks,

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 04-08-2011 20:22, Matthew Dharm escreveu:
> On Thu, Aug 4, 2011 at 3:56 PM, Greg KH <greg@kroah.com <mailto:greg@kroah.com>> wrote:
> 
>     On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>     > I know that this problem were somewhat solved for 3G modems, with the usage
>     > of the userspace problem usb_modeswitch, and with some quirks for the USB
>     > storage driver, but I'm not sure if such tricks will scale forever, as more
>     > functions are seen on some USB devices.
> 
>     Well, no matter how it "scales" it needs to be done in userspace, like
>     usb_modeswitch does.  We made that decision a while ago, and it is
>     working out very well.  I see no reason why you can't do it in userspace
>     as well as that is the easiest place to control this type of thing.
> 
>     I thought we had a long discussion about this topic a while ago and came
>     to this very conclusion.  Or am I mistaken?
> 
>  
> We keep having the discussion over and over again.  But, you are correct: the conclusion was that this all needs to live in userspace.

In the case of 3G modem x USB storage only, it is possible to handle it on userspace.

However, when there are more functions added, an they're not (completely) mutually exclusive,
then I don't see an easy way (if is there a way) for doing it at userspace.

Several devices offer more than one function at the same time, but some
resources are mutually exclusive. A TV stick with just one tuner, and
both analog and digital demods offer both analog and digital TV at the
same time. So, both analog and digital parts of the driver will offer
the device to userspace. However, unpredictable results will happen if
userspace tries to use both at the same time.

The Digital camera devices that offer PTP transfers and V4L support also
fall at the same type of trouble. Some of those devices just delete
all pictures from the memory, if streaming is started. So, receiving
an automatic Skype video call may delete all pictures you took.

Worse than that, currently, the PTP protocol is handled via libusb, while
streaming is done via V4L2 API.

The best technical approach, IMO, is to implement the PTP protocol in
kernelspace, and do some sort of inter-subsystem locking to prevent such
troubles.

Regards,
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Thu, 4 Aug 2011, Greg KH wrote:

> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > I know that this problem were somewhat solved for 3G modems, with the usage
> > of the userspace problem usb_modeswitch, and with some quirks for the USB
> > storage driver, but I'm not sure if such tricks will scale forever, as more
> > functions are seen on some USB devices.
> 
> Well, no matter how it "scales" it needs to be done in userspace, like
> usb_modeswitch does.  We made that decision a while ago, and it is
> working out very well.  I see no reason why you can't do it in userspace
> as well as that is the easiest place to control this type of thing.
> 
> I thought we had a long discussion about this topic a while ago and came
> to this very conclusion.  Or am I mistaken?
> 
> thanks,
> 
> greg k-h
> 

Greg,

A little bit more of precision would help me a lot, here. Precisely what 
is one supposed to do in userspace? I would naturally assume that the user 
will control the device by announcing what the user wants to do with the 
device. Such control would therefore have to start in userspace. But this 
is such an obvious truism that you must mean something else. What, then?

To say to do things "like usb_modeswitch" is rather vague. For one thing, 
usb_modeswitch, being something out there in userspace, does _not_ affect 
what is in the kernel. However, what is (or is not) in the kernel might 
make it possible (or, in the alternative, impossible) to facilitate the 
action of some kind of userspace function-switching program similar to 
usb_modeswitch, not so? 

As to the "long discussion about this topic a while ago" that may have 
been a discussion in which I was also involved. For, I do remember a 
discussion about this topic a few months ago. In one respect my memory 
differs from yours, however: It was not my impression that any definite 
conclusions were reached, certainly not a consensus. BTW, I did try to lay 
out what I remembered as three alternatives that came up, along with 
arguments for and against each of them, in my message which kicked off 
this thread today. 

It is of course a real possibility that you have seen the perfect solution 
in your mind to these problems, which have vexed a lot of people for 
years, and others have not. If that is the way it is, then perhaps you 
just wish that we would all shut up, implement what is obvious, and we 
could all be happy. This could certainly be the case. You have written a 
lot more code than I have, and you obviously must have started doing that 
when you were at least 30 years younger, perhaps even 40 years younger, 
than I was when I started. So I would be the first to say that you are 
much better at it than I am. But there are others here, too, who seem 
actively to have been sucked into this discussion, and they are similarly 
younger and more clever than I am, too. 

Giving due consideration to this, it would probably get the job done a lot 
faster if you just take the time to describe what grand vision you 
yourself have in mind for the solution of the problem, with a sufficient 
accounting of the details that people can all see what it is and exactly 
how and why it would work absolutely perfectly, and then we can just get 
busy and do it.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Oliver Neukum]

Am Freitag, 5. August 2011, 00:56:03 schrieb Greg KH:
> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > I know that this problem were somewhat solved for 3G modems, with the usage
> > of the userspace problem usb_modeswitch, and with some quirks for the USB
> > storage driver, but I'm not sure if such tricks will scale forever, as more
> > functions are seen on some USB devices.
> 
> Well, no matter how it "scales" it needs to be done in userspace, like
> usb_modeswitch does.  We made that decision a while ago, and it is
> working out very well.  I see no reason why you can't do it in userspace
> as well as that is the easiest place to control this type of thing.
> 
> I thought we had a long discussion about this topic a while ago and came
> to this very conclusion.  Or am I mistaken?

Circumstances change. We want to keep the stuff in user space as much and
as long as we can. However user space has limitations:

- it has by necessity a race between resumption and access by others
- it cannot resume anything we run a (rw) filesystem over.

Furthermore, today PM actions that lead to a loss of mode are initiated
by user space. If we ever want to oportunistically suspend a system
we also need to restore mode from inside the kernel.

We could avoid all that trouble, if we persuaded vendors to use plain
USB configurations for those purposes.

	Regards
		Oliver

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/05/2011 12:56 AM, Greg KH wrote:
> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>> I know that this problem were somewhat solved for 3G modems, with the usage
>> of the userspace problem usb_modeswitch, and with some quirks for the USB
>> storage driver, but I'm not sure if such tricks will scale forever, as more
>> functions are seen on some USB devices.
>
> Well, no matter how it "scales" it needs to be done in userspace, like
> usb_modeswitch does.  We made that decision a while ago, and it is
> working out very well.  I see no reason why you can't do it in userspace
> as well as that is the easiest place to control this type of thing.
>
> I thought we had a long discussion about this topic a while ago and came
> to this very conclusion.  Or am I mistaken?

I think we've had multiple discussions about this, surrounding various
topics / use cases. I would not call the do it in userspace a conclusion.

I rather more have a feeling of getting stonewalled on this by various people
surrounding the usb system. Me shutting up on this is basically a case of:
"Discussion not getting anywhere and I don't have the time to do a kernel
  proof of concept myself right now".

To be clear about the stonewalling I'm not talking about the dual cam issue
here, nor about the morphing devices thing which usb_modeswitch fixes.

I think it is important to separate oranges from apples here, there are
at least 3 different problem classes which all seem to have gotten thrown
onto a pile here:

1) The reason Mauro suggested having some discussion on this at the
USB summit is because of a discussion about dual mode cameras on the
linux media list. Dual-mode cameras are (usually very cheap) digital
photo cameras which can take still pictures in stand alone mode and
store them in onboard memory, like regular digital photo cameras, this
is one mode. The other mode is they can operate as a regular webcam,
the hardware actually is more regular webcam hardware with some memory
and a battery added (usually no viewfinder screen).

These devices typically use 1 usb interface (and thus 1 driver) for
both modes. We currently have drivers for both modes, but they are
separate drivers, which leads to fighting for device ownership,
with the stillmode driver always winning as that is userspace and
it simply kicks of the kernel webcam /streaming mode driver making
it see a device unplug, even if an app was streaming from the device
at the time. The solution here is simple, move both functions to
1 driver, which can then properly return -EBUSY when the device
is active in one mode and an app tries to use it in the other mode.

This 1 driver will likely end up being a kernel driver, since doing
streaming drivers in userspace is problematic, as the v4l2 API assumes
a kernel device node, and we don't want to invent a new API, we want
something which works with existing applications.

2) So called morphing devices. For example a USB 3G modem which
initially shows up as a cdrom drive with windows software, only
to reveal its real identity (with completely different usb
descriptors) after some magic command. This is solved by the
usb_modeswitch command, nothing to see here move along.

3) Re-direction of usb devices to virtual machines. This works by using
the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
The basics of this work fine, but it lacks proper locking / safeguards
for when a vm takes over a usb device from the in kernel driver.

Example of the problem: usb mass storage devices is mounted, filesystem
is in use and dirty, with writebacks pending, user redirects to vm,
usb mass storage driver sees the cable beeing yanked out of the device.

Now we've lost the pending writebacks and have a dirty, if not journaling
possibly corrupt, fs on the device.

This is the issue on which I feel a bit stonewalled. Simple putting your
fingers in your ears and singing la la la do it in userspace is not going
to cut it here. There is no way to do this race free in userspace, unless
all possible callers of mount get modified. Moreover 99% of the necessary
accounting for this is already done in the kernel. We already have the notion
of a block device being in use. We simply need to add some code to pass
this notion to the usb mass storage driver, and add a new try_disconnect
callback for usb drivers. I'm not saying this is going to be completely
straight forward, but it ain't rocket science either. And it so very
obviously is the *right* thing to do, that I'm getting very tired of
the do it in userspace song I keep hearing.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouveroliver

[Oliver Neukum]

Am Freitag, 5. August 2011, 09:45:56 schrieb Hans de Goede:
> This is the issue on which I feel a bit stonewalled. Simple putting your
> fingers in your ears and singing la la la do it in userspace is not going
> to cut it here. There is no way to do this race free in userspace, unless
> all possible callers of mount get modified. Moreover 99% of the necessary
> accounting for this is already done in the kernel. We already have the notion
> of a block device being in use. We simply need to add some code to pass
> this notion to the usb mass storage driver, and add a new try_disconnect
> callback for usb drivers. I'm not saying this is going to be completely
> straight forward, but it ain't rocket science either. And it so very
> obviously is the right thing to do, that I'm getting very tired of
> the do it in userspace song I keep hearing.

Doing a try_disconnect() would also solve the dual camera issue.
But it doesn't really interfere with the user space vs. kernel space
issue. You simply have to expand the ioctl interface to have
an ioctl that triggers this API call in the kernel.

A V4L2 device would return an error if its device node is opened,
otherwise disconnect.

	Regards
		Oliver

		

Re: USB mini-summit at LinuxCon Vancouveroliver

[Hans de Goede]

Hi,

On 08/05/2011 09:59 AM, Oliver Neukum wrote:
> Am Freitag, 5. August 2011, 09:45:56 schrieb Hans de Goede:
>> This is the issue on which I feel a bit stonewalled. Simple putting your
>> fingers in your ears and singing la la la do it in userspace is not going
>> to cut it here. There is no way to do this race free in userspace, unless
>> all possible callers of mount get modified. Moreover 99% of the necessary
>> accounting for this is already done in the kernel. We already have the notion
>> of a block device being in use. We simply need to add some code to pass
>> this notion to the usb mass storage driver, and add a new try_disconnect
>> callback for usb drivers. I'm not saying this is going to be completely
>> straight forward, but it ain't rocket science either. And it so very
>> obviously is the right thing to do, that I'm getting very tired of
>> the do it in userspace song I keep hearing.
>
> Doing a try_disconnect() would also solve the dual camera issue.
> But it doesn't really interfere with the user space vs. kernel space
> issue. You simply have to expand the ioctl interface to have
> an ioctl that triggers this API call in the kernel.
>
> A V4L2 device would return an error if its device node is opened,
> otherwise disconnect.

Getting a bit offtopic here, but no a try_disconnect will fix the
userspace stillcam mode driver being able to disconnect the device
while the webcam function is active. If the webcam is not active
userspace will still "win", and possibly never return the device
back to the kernel driver (this already happens today with
gvfs-gphoto creating a fuse mount and keeping the device open
indefinitely, locking out the webcam function).

Likewise a v4l2 control panel like app (think alsamixer for
videodevs to set brightness / contrast, etc.) can keep the /dev/video
node open indefinitely. Unless we rewrite most of userspace, we need
to allow the device to be open in bode modes *at the same time* and
only fail with -EBUSY when something really exclusive is requested
(so not just having the device open, or setting contrast, but
trying to stream and read/delete pictures from the stillcam
memory at the same time).

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 05-08-2011 04:45, Hans de Goede escreveu:
> Hi,
> 
> On 08/05/2011 12:56 AM, Greg KH wrote:
>> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>>> I know that this problem were somewhat solved for 3G modems, with the usage
>>> of the userspace problem usb_modeswitch, and with some quirks for the USB
>>> storage driver, but I'm not sure if such tricks will scale forever, as more
>>> functions are seen on some USB devices.
>>
>> Well, no matter how it "scales" it needs to be done in userspace, like
>> usb_modeswitch does.  We made that decision a while ago, and it is
>> working out very well.  I see no reason why you can't do it in userspace
>> as well as that is the easiest place to control this type of thing.
>>
>> I thought we had a long discussion about this topic a while ago and came
>> to this very conclusion.  Or am I mistaken?
> 
> I think we've had multiple discussions about this, surrounding various
> topics / use cases. I would not call the do it in userspace a conclusion.
> 
> I rather more have a feeling of getting stonewalled on this by various people
> surrounding the usb system. Me shutting up on this is basically a case of:
> "Discussion not getting anywhere and I don't have the time to do a kernel
>  proof of concept myself right now".
> 
> To be clear about the stonewalling I'm not talking about the dual cam issue
> here, nor about the morphing devices thing which usb_modeswitch fixes.
> 
> I think it is important to separate oranges from apples here, there are
> at least 3 different problem classes which all seem to have gotten thrown
> onto a pile here:
> 
> 1) The reason Mauro suggested having some discussion on this at the
> USB summit is because of a discussion about dual mode cameras on the
> linux media list. Dual-mode cameras are (usually very cheap) digital
> photo cameras which can take still pictures in stand alone mode and
> store them in onboard memory, like regular digital photo cameras, this
> is one mode. The other mode is they can operate as a regular webcam,
> the hardware actually is more regular webcam hardware with some memory
> and a battery added (usually no viewfinder screen).

Yes, that's my concern.

> These devices typically use 1 usb interface (and thus 1 driver) for
> both modes. 

I also want to cover the case where there are two drivers involved, as
this is a typical trouble we need to do with all devices that have both
analog and digital TV's.

A typical media device exposes several different API's to userspace,
each with its own character device, and sometimes implemented by a
different driver.

For example, Conexant cx23102 USB chipsets provide 2 USB interfaces,
but require 3 different drivers, and exposes 4 different userspace
API's to userspace:

USB interface 0: Remote Controller
	event/input API - Handled by drivers/media/rc/mceusb.c

USB interface 1: Audio/Video
	ALSA API - Handled by drivers/media/video/cx231xx/cx231xx-audio.c
		It is a separate driver, called cx231xx-alsa
	V4L2 API - Handled by drivers/media/video/cx231xx/cx231xx-video.c
	DVB API - Handled by drivers/media/video/cx231xx/cx231xx-dvb.c
		Both DVB and V4L2 are currently part of the same driver
(cx231xx).

The mceusb is completely independent of the other drivers. The cx231xx-alsa
also is almost independent. It only shares a lock with the main cx231xx
driver, and uses the I2C functions implemented by the main driver.

Now, we're starting to see devices like ZTE MF845 that have TV, 3G and 
USB storage, exposing 2 more userspace API's to userspace (vfs and tty).

I don't think that userspace usb_modeswitch-like type of locking "scales"
such complex devices.

Besides that, on a large amount of USB sticks, if you enable both V4L2
and DVB, the power consumption may exceed the 500mA power limit of USB,
and can overheat the device. There are several reports of burnt devices
due to that. Also, there's no way for both to work, as they share the
same tuner, that can either be tuning an analog or a digital channel.

To me, the right solution seems to add some sort of locking schema that
will protect the common resources on complex hardware. Something like:

int get_hw_resource(struct device, enum hw_resource_type);
void put_hw_resource(struct device, enum hw_resource_type);

enum hw_resource_type {
	HW_RES_USB,
	HW_RES_I2C,
	HW_RES_TUNER,
	HW_RES_USB,
...
};

So that, when a device need, for example, an exclusive lock for the USB
device, for example, it could be doing something like:

rc = get_hw_resource(dev, HW_RES_USB);
if (rc < 0)
	return -EBUSY;

and, after finishing, doing a put_hw_resource().

> We currently have drivers for both modes, but they are
> separate drivers, which leads to fighting for device ownership,
> with the stillmode driver always winning as that is userspace and
> it simply kicks of the kernel webcam /streaming mode driver making
> it see a device unplug, even if an app was streaming from the device
> at the time. The solution here is simple, move both functions to
> 1 driver, which can then properly return -EBUSY when the device
> is active in one mode and an app tries to use it in the other mode.
> 
> This 1 driver will likely end up being a kernel driver, since doing
> streaming drivers in userspace is problematic, as the v4l2 API assumes
> a kernel device node, and we don't want to invent a new API, we want
> something which works with existing applications.
> 
> 2) So called morphing devices. For example a USB 3G modem which
> initially shows up as a cdrom drive with windows software, only
> to reveal its real identity (with completely different usb
> descriptors) after some magic command. This is solved by the
> usb_modeswitch command, nothing to see here move along.
> 
> 3) Re-direction of usb devices to virtual machines. This works by using
> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> The basics of this work fine, but it lacks proper locking / safeguards
> for when a vm takes over a usb device from the in kernel driver.
> 
> Example of the problem: usb mass storage devices is mounted, filesystem
> is in use and dirty, with writebacks pending, user redirects to vm,
> usb mass storage driver sees the cable beeing yanked out of the device.
> 
> Now we've lost the pending writebacks and have a dirty, if not journaling
> possibly corrupt, fs on the device.
> 
> This is the issue on which I feel a bit stonewalled. Simple putting your
> fingers in your ears and singing la la la do it in userspace is not going
> to cut it here. There is no way to do this race free in userspace, unless
> all possible callers of mount get modified. Moreover 99% of the necessary
> accounting for this is already done in the kernel. We already have the notion
> of a block device being in use. We simply need to add some code to pass
> this notion to the usb mass storage driver, and add a new try_disconnect
> callback for usb drivers. I'm not saying this is going to be completely
> straight forward, but it ain't rocket science either. And it so very
> obviously is the *right* thing to do, that I'm getting very tired of
> the do it in userspace song I keep hearing.
> 
> Regards,
> 
> Hans

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Fri, 5 Aug 2011, Oliver Neukum wrote:

> Am Freitag, 5. August 2011, 00:56:03 schrieb Greg KH:
> > On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > > I know that this problem were somewhat solved for 3G modems, with the usage
> > > of the userspace problem usb_modeswitch, and with some quirks for the USB
> > > storage driver, but I'm not sure if such tricks will scale forever, as more
> > > functions are seen on some USB devices.
> > 
> > Well, no matter how it "scales" it needs to be done in userspace, like
> > usb_modeswitch does.  We made that decision a while ago, and it is
> > working out very well.  I see no reason why you can't do it in userspace
> > as well as that is the easiest place to control this type of thing.
> > 
> > I thought we had a long discussion about this topic a while ago and came
> > to this very conclusion.  Or am I mistaken?
> 
> Circumstances change. We want to keep the stuff in user space as much and
> as long as we can. However user space has limitations:
> 
> - it has by necessity a race between resumption and access by others
> - it cannot resume anything we run a (rw) filesystem over.
> 
> Furthermore, today PM actions that lead to a loss of mode are initiated
> by user space. If we ever want to oportunistically suspend a system
> we also need to restore mode from inside the kernel.
> 
> We could avoid all that trouble, if we persuaded vendors to use plain
> USB configurations for those purposes.

But that would happen, I suspect, in an alternate universe. Better to 
anticipate the trouble, I suspect. :-{

Moreover, the spark for the current discussion was the problem of 
dual-mode cameras, which can work both as webcams and stillcams, not the 
3G modems that you mention. The problems are analogous but not identical. 

	-- dual-mode cameras are, typically, Class Proprietary devices in 
all of their functions. None of them that I know of are Mass Storage 
devices. Therefore, usb_modeswitch would have to be rewritten completely 
in order to be used for such hardware. As things stand right now, it has 
nothing at all to do with the problem. Not to say, of course, that the 
experience gained with usb_modeswitch is totally irrelevant.

	-- I don't have one of those modems, but I have the impression 
that the need to access the "disk" partition on one of them is basically a 
one-shot deal. Namely, one needs to load some firmware or so on the "disk" 
before the gadget can be used. The problem with a dual-mode camera is that 
the user ought to be able to switch at will between a 
download-the-pictures stillcam application and a stream application. 
While, of course, not being able to start one of these activities while 
the other is going on, because that would cause obvious trouble. 

The similarity of the modem-and-mass-storage device and the web-and-still 
camera is, of course, that both partake of being devices that will do more 
than one kind of thing and they need to be supported in that respect. 
Other hardware exists with similar characteristics, but sometimes the 
functionaliy is not dual but even triple, and one can reasonably suspect 
that more of this kind of thing is going to come at us in the future. I 
think it is a good occasion to sit back and think things over a bit.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> Hi,
> 
> On 08/05/2011 12:56 AM, Greg KH wrote:
> >On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> I think it is important to separate oranges from apples here, there are
> at least 3 different problem classes which all seem to have gotten thrown
> onto a pile here:
> 
> 1) The reason Mauro suggested having some discussion on this at the
> USB summit is because of a discussion about dual mode cameras on the
> linux media list.
...
> 3) Re-direction of usb devices to virtual machines. This works by using
> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> The basics of this work fine, but it lacks proper locking / safeguards
> for when a vm takes over a usb device from the in kernel driver.

Hi Hans and Mauro,

We have do room in the schedule for the USB mini-summit for this
discussion, since the schedule is still pretty flexible.  The
preliminary schedule is up here:

http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html

I think it's best to discuss the VM redirection in the afternoon when
some of the KVM folks join us after Hans' talk on USB redirection over
the network.

It sounds like we need a separate topic for the dual mode cameras and TV
tuners.  Mauro, do you want to lead that discussion in the early morning
(in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
slot)?  I want to be sure we have all the video/media developers who are
interested in this topic present, and I don't know if they will be going
to the KVM forum.

Link PM was originally slated for the 10am slot, but since we're missing
several people for that discussion (Alan Stern, Andiry Xu, and any of
the Intel folks who did the Moorestown USB 2.0 LPM), I think it will be
more useful to have that discussion on-list or at another conference.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Mon, 8 Aug 2011, Sarah Sharp wrote:

> On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> > Hi,
> > 
> > On 08/05/2011 12:56 AM, Greg KH wrote:
> > >On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > I think it is important to separate oranges from apples here, there are
> > at least 3 different problem classes which all seem to have gotten thrown
> > onto a pile here:
> > 
> > 1) The reason Mauro suggested having some discussion on this at the
> > USB summit is because of a discussion about dual mode cameras on the
> > linux media list.
> ...
> > 3) Re-direction of usb devices to virtual machines. This works by using
> > the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> > The basics of this work fine, but it lacks proper locking / safeguards
> > for when a vm takes over a usb device from the in kernel driver.
> 
> Hi Hans and Mauro,
> 
> We have do room in the schedule for the USB mini-summit for this
> discussion, since the schedule is still pretty flexible.  The
> preliminary schedule is up here:
> 
> http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> 
> I think it's best to discuss the VM redirection in the afternoon when
> some of the KVM folks join us after Hans' talk on USB redirection over
> the network.
> 
> It sounds like we need a separate topic for the dual mode cameras and TV
> tuners.  Mauro, do you want to lead that discussion in the early morning
> (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> slot)?  I want to be sure we have all the video/media developers who are
> interested in this topic present, and I don't know if they will be going
> to the KVM forum.

Sarah,

Alas. I would suspect that I am one of the people most interested in the 
topic of dual-mode cameras, since I have worked on supporting them both in 
libgphoto2 and in the kernel. But I teach in a university for a living, 
and the first classes of Fall Semester 2011 start on August 17 in Auburn, 
Alabama. Knowing this, I decided, months ago, that I simply could not 
attend a conference which starts on August 16 in Vancouver.

So, after starting all of the current mailing-list discussion on the topic 
I will not be at the conference. I can only hope that those who do attend 
will keep me current about what gets discussed.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Mon, Aug 08, 2011 at 01:23:56PM -0500, Theodore Kilgore wrote:
> 
> 
> On Mon, 8 Aug 2011, Sarah Sharp wrote:
> 
> > On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> > > Hi,
> > > 
> > > On 08/05/2011 12:56 AM, Greg KH wrote:
> > > >On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > > I think it is important to separate oranges from apples here, there are
> > > at least 3 different problem classes which all seem to have gotten thrown
> > > onto a pile here:
> > > 
> > > 1) The reason Mauro suggested having some discussion on this at the
> > > USB summit is because of a discussion about dual mode cameras on the
> > > linux media list.
> > ...
> > > 3) Re-direction of usb devices to virtual machines. This works by using
> > > the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> > > The basics of this work fine, but it lacks proper locking / safeguards
> > > for when a vm takes over a usb device from the in kernel driver.
> > 
> > Hi Hans and Mauro,
> > 
> > We have do room in the schedule for the USB mini-summit for this
> > discussion, since the schedule is still pretty flexible.  The
> > preliminary schedule is up here:
> > 
> > http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> > 
> > I think it's best to discuss the VM redirection in the afternoon when
> > some of the KVM folks join us after Hans' talk on USB redirection over
> > the network.
> > 
> > It sounds like we need a separate topic for the dual mode cameras and TV
> > tuners.  Mauro, do you want to lead that discussion in the early morning
> > (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> > slot)?  I want to be sure we have all the video/media developers who are
> > interested in this topic present, and I don't know if they will be going
> > to the KVM forum.
> 
> Sarah,
> 
> Alas. I would suspect that I am one of the people most interested in the 
> topic of dual-mode cameras, since I have worked on supporting them both in 
> libgphoto2 and in the kernel. But I teach in a university for a living, 
> and the first classes of Fall Semester 2011 start on August 17 in Auburn, 
> Alabama. Knowing this, I decided, months ago, that I simply could not 
> attend a conference which starts on August 16 in Vancouver.
> 
> So, after starting all of the current mailing-list discussion on the topic 
> I will not be at the conference. I can only hope that those who do attend 
> will keep me current about what gets discussed.

No worries, I'll be taking notes and post them to the Linux USB and
Linux media lists.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Hi Sarah,

Em 08-08-2011 14:58, Sarah Sharp escreveu:
> On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 08/05/2011 12:56 AM, Greg KH wrote:
>>> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>> I think it is important to separate oranges from apples here, there are
>> at least 3 different problem classes which all seem to have gotten thrown
>> onto a pile here:
>>
>> 1) The reason Mauro suggested having some discussion on this at the
>> USB summit is because of a discussion about dual mode cameras on the
>> linux media list.
> ...
>> 3) Re-direction of usb devices to virtual machines. This works by using
>> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
>> The basics of this work fine, but it lacks proper locking / safeguards
>> for when a vm takes over a usb device from the in kernel driver.
> 
> Hi Hans and Mauro,
> 
> We have do room in the schedule for the USB mini-summit for this
> discussion, since the schedule is still pretty flexible.  The
> preliminary schedule is up here:
> 
> http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> 
> I think it's best to discuss the VM redirection in the afternoon when
> some of the KVM folks join us after Hans' talk on USB redirection over
> the network.
> 
> It sounds like we need a separate topic for the dual mode cameras and TV
> tuners.  

Yes, I suspect so. This subject is pending for quite some time, and it would
be good to have people think together around a solution for that.

> Mauro, do you want to lead that discussion in the early morning
> (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> slot)?  I want to be sure we have all the video/media developers who are
> interested in this topic present, and I don't know if they will be going
> to the KVM forum.

>From my side, both slots work ok. Hans, what would fit better for you?

> Link PM was originally slated for the 10am slot, but since we're missing
> several people for that discussion (Alan Stern, Andiry Xu, and any of
> the Intel folks who did the Moorestown USB 2.0 LPM), I think it will be
> more useful to have that discussion on-list or at another conference.
> 
> Sarah Sharp

Thanks!
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/08/2011 07:58 PM, Sarah Sharp wrote:
> On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 08/05/2011 12:56 AM, Greg KH wrote:
>>> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>> I think it is important to separate oranges from apples here, there are
>> at least 3 different problem classes which all seem to have gotten thrown
>> onto a pile here:
>>
>> 1) The reason Mauro suggested having some discussion on this at the
>> USB summit is because of a discussion about dual mode cameras on the
>> linux media list.
> ...
>> 3) Re-direction of usb devices to virtual machines. This works by using
>> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
>> The basics of this work fine, but it lacks proper locking / safeguards
>> for when a vm takes over a usb device from the in kernel driver.
>
> Hi Hans and Mauro,
>
> We have do room in the schedule for the USB mini-summit for this
> discussion, since the schedule is still pretty flexible.  The
> preliminary schedule is up here:
>
> http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
>
> I think it's best to discuss the VM redirection in the afternoon when
> some of the KVM folks join us after Hans' talk on USB redirection over
> the network.
>

That seems best to me too. I'm available at both proposed time slots,
and I would like to join both discussions.

> It sounds like we need a separate topic for the dual mode cameras and TV
> tuners.  Mauro, do you want to lead that discussion in the early morning
> (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> slot)?  I want to be sure we have all the video/media developers who are
> interested in this topic present, and I don't know if they will be going
> to the KVM forum.

I would really like to see the dual mode camera and TV tuner discussion
separated. They are 2 different issues AFAIK.

1) Dual mode cameras:

In the case of the dual mode camera we have 1 single device (both at
the hardware level and at the logical block level), which can do 2 things,
but not at the same time. It can stream live video data from a sensor,
or it can retrieve earlier taken pictures from some picture memory.

Unfortunately even though these 2 functions live in a single logical block,
historically we've developed 2 drivers for them. This leads to fighting
over device ownership (surprise surprise), and to me the solution is
very clear, 1 logical block == 1 driver.

2) Tv tuners:

Here we have a bunch of logical blocks, each with their own driver
(and not 2 drivers for one block), which together form not 1 but 2
video pipelines, typically one pipeline for analog TV, and one
for DVB. The problem here is some blocks are shared between the
2 pipelines.

The solution here, at least to me, is clear too, we need some way
of claiming/locking the pipeline blocks. If pipe1 starts streaming
video data it locks all the blocks it uses. If some app then tries
to use pipe2 (and thus for example tune the tuner to a different
frequency), it will fail with -EBUSY when pipe2 tries to take the
tuner lock and gets told it is in use.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Tue, 9 Aug 2011, Hans de Goede wrote:

> I would really like to see the dual mode camera and TV tuner discussion
> separated. They are 2 different issues AFAIK.
> 
> 1) Dual mode cameras:
> 
> In the case of the dual mode camera we have 1 single device (both at
> the hardware level and at the logical block level), which can do 2 things,
> but not at the same time. It can stream live video data from a sensor,
> or it can retrieve earlier taken pictures from some picture memory.
> 
> Unfortunately even though these 2 functions live in a single logical block,
> historically we've developed 2 drivers for them. This leads to fighting
> over device ownership (surprise surprise), and to me the solution is
> very clear, 1 logical block == 1 driver.

According to Theodore, we have developed 5 drivers for them because the
stillcam modes in different devices use four different vendor-specific
drivers.  Does it really make sense to combine 5 drivers into one?  I 
think some sort of sharing protocol would work out better.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Marko Ristola]

Hi

I've been thinking about the Kernel driver side.
Mauro and others emailed requirements on Jun or July.

I'm sorry for this spam: maybe you have thought this already.

A linked list of read/write locks as the solution for these protections could be
a base for the general solution. Locks could be accessed either by name "string name"
or by an integer identifier.

The bridge driver would be the container of the lock list.
Lock take / release handles would be changeable by Bridge driver at driver init.
This way bridge could tune the lock taking actions. Sub devices would not have
to know the details of the bridge device.

When implemented as a library ".ko" module, this could be
used by all related kernel drivers. The locking code would be very general.
Maybe adding a lock list into each PCI bus device would solve the device export problem to KVM too.

If some module wouldn't handle the proper locking yet,
it would not deliver the protection, but it would work as before: no regressions.

The library could also be called so that a driver would ask three locks at the same time.
If the driver would get all three locks, it would return success.
If the driver would not get all three locks, it would not lock any of them (with _trylock case).

I don't have time to implement this feature.

Happy meeting for all of you,
Marko Ristola

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Tue, Aug 09, 2011 at 09:52:44AM +0200, Hans de Goede wrote:
> Hi,
> 
> On 08/08/2011 07:58 PM, Sarah Sharp wrote:
> >On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> >>Hi,
> >>
> >>On 08/05/2011 12:56 AM, Greg KH wrote:
> >>>On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> >>I think it is important to separate oranges from apples here, there are
> >>at least 3 different problem classes which all seem to have gotten thrown
> >>onto a pile here:
> >>
> >>1) The reason Mauro suggested having some discussion on this at the
> >>USB summit is because of a discussion about dual mode cameras on the
> >>linux media list.
> >...
> >>3) Re-direction of usb devices to virtual machines. This works by using
> >>the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> >>The basics of this work fine, but it lacks proper locking / safeguards
> >>for when a vm takes over a usb device from the in kernel driver.
> >
> >Hi Hans and Mauro,
> >
> >We have do room in the schedule for the USB mini-summit for this
> >discussion, since the schedule is still pretty flexible.  The
> >preliminary schedule is up here:
> >
> >http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> >
> >I think it's best to discuss the VM redirection in the afternoon when
> >some of the KVM folks join us after Hans' talk on USB redirection over
> >the network.
> >
> 
> That seems best to me too. I'm available at both proposed time slots,
> and I would like to join both discussions.
> 
> >It sounds like we need a separate topic for the dual mode cameras and TV
> >tuners.  Mauro, do you want to lead that discussion in the early morning
> >(in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> >slot)?  I want to be sure we have all the video/media developers who are
> >interested in this topic present, and I don't know if they will be going
> >to the KVM forum.
> 
> I would really like to see the dual mode camera and TV tuner discussion
> separated. They are 2 different issues AFAIK.

Ok, great, I've put both topics in the morning, in separate slots.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/09/2011 04:19 PM, Alan Stern wrote:
> On Tue, 9 Aug 2011, Hans de Goede wrote:
>
>> I would really like to see the dual mode camera and TV tuner discussion
>> separated. They are 2 different issues AFAIK.
>>
>> 1) Dual mode cameras:
>>
>> In the case of the dual mode camera we have 1 single device (both at
>> the hardware level and at the logical block level), which can do 2 things,
>> but not at the same time. It can stream live video data from a sensor,
>> or it can retrieve earlier taken pictures from some picture memory.
>>
>> Unfortunately even though these 2 functions live in a single logical block,
>> historically we've developed 2 drivers for them. This leads to fighting
>> over device ownership (surprise surprise), and to me the solution is
>> very clear, 1 logical block == 1 driver.
>
> According to Theodore, we have developed 5 drivers for them because the
> stillcam modes in different devices use four different vendor-specific
> drivers.

Yes, but so the the webcam modes of the different devices, so for
the 5 (not sure if that is the right number) dual-cam mode chipsets
we support there will be 5 drivers, each supporting both the
webcam and the access to pictures stored in memory of the chipset
they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
and both functions of the single logical device that chipset
represents.

>  Does it really make sense to combine 5 drivers into one?

Right, that is not the plan. The plan is to simply stop having 2 drivers
for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
+ 5 webcam, to just 5 drivers. We will also likely be able to share
code between the code for the 2 functionalities for things like generic
set / get register functions, initialization, etc.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Adam Baker]

On Tuesday 09 August 2011, Hans de Goede wrote:
> Hi,
> 
> On 08/09/2011 04:19 PM, Alan Stern wrote:

> >  Does it really make sense to combine 5 drivers into one?
> 
> Right, that is not the plan. The plan is to simply stop having 2 drivers
> for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> + 5 webcam, to just 5 drivers. We will also likely be able to share
> code between the code for the 2 functionalities for things like generic
> set / get register functions, initialization, etc.
> 

Unfortunately as Theodore recently pointed out you don't go from 10 to 5, you 
go from 10 to 10 where 5 of the new 10 are only used on Win32, FreeBSD and OSX 
(but they aren't any simpler because they still rely on libusb) and the other 
5 that are only used on Linux become significantly more complicated than they 
currently are.

It has also just occured to me that it might be possible to solve the issues 
we are facing just in the kernel. At the moment when the kernel performs a 
USBDEVFS_DISCONNECT it keeps the kernel driver locked out until userspace 
performs a USBDEVFS_CONNECT. If the kernel reattached the kernel driver when 
the device file was closed then, as gvfs doesn't keep the file open the 
biggest current issue would be solved instantly. If a mechanism could be found 
to prevent USBDEVFS_DISCONNECT from succeeding when the corresponding 
/dev/videox file was open then that would seem to be a reasonable solution.

Hans had expressed the opinion that merely having the device open to control 
the camera not to stream shouldn't  prevent stillcam operation - I disagree 
because if you are setting up the controls you are probably already streaming 
so you can see what you are doing and if not you are probably about to.

Of course changing the behaviour of USBDEVFS_DISCONNECT is not something to be 
done lightly. I don't know how many other users there are for it and if the 
current behaviour is actually correct for any of them. Cleaning up on file 
close does have the useful side effect though that applications no longer need 
to worry about the fact that even if they clean up properly on a normal exit, 
if they crash they leave the kernel driver permanently disabled

Regards

Adam

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/09/2011 10:31 PM, Adam Baker wrote:
> On Tuesday 09 August 2011, Hans de Goede wrote:

<snip>

> It has also just occured to me that it might be possible to solve the issues
> we are facing just in the kernel. At the moment when the kernel performs a
> USBDEVFS_DISCONNECT it keeps the kernel driver locked out until userspace
> performs a USBDEVFS_CONNECT. If the kernel reattached the kernel driver when
> the device file was closed then, as gvfs doesn't keep the file open the
> biggest current issue would be solved instantly. If a mechanism could be found
> to prevent USBDEVFS_DISCONNECT from succeeding when the corresponding
> /dev/videox file was open then that would seem to be a reasonable solution.

<sigh>

This has been discussed over and over and over again, playing clever
tricks with USBDEVFS_[DIS]CONNECT like adding a new USBDEVFS_TRYDISCONNECT
which the v4l2 driver could intercept won't cut it. We need some central
manager of the device doing multiplexing between the 2 functions, and you
can *not* assume that either side will be nice wrt closing file descriptors.

Examples:
1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the
time, and through that the usbfs device nodes.

2) Lets say a user starts a photo managing app like f-spot, and that opens
the device through libgphoto2 on startup, then the user switches to another
virtual desktop and forgets all about having f-spot open. Notice that if
the user now tries to stream he will not get a busy error, but the app trying
to do the streaming will simply not see the camera at all (kernel driver
unbound /dev/video# node is gone).

3) Notice that little speaker icon in your panel on your average Linux
desktop, that keeps the mixer of the audio device open *all the time* it
is quite easy to imagine a similar applet for v4l2 device controls (see
for example gtk-v4l) doing the same. Or a user could simply start up a
v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running
minimized ...

4) Some laptops have a Fn + F## key which enables / disables the builtin
webcam by turning its power on / off. Effectively plugging it into / out
of a usb port. We would like to have an on screen notification of this one
day like we have now for brightness and volume controls, based on udev
events. But the current dual mode cam stuff causes udev events for
a *new* video device being added / an existing one being removed
each time libgphoto2 releases / takes control of the camera.

5) More in general, more and more software is dynamically monitoring the
addition / removal of (usb) devices using udev, our current solution
suggests to this software the /dev/video device is being unplugged /
re-plugged all the time, not pretty.


All in all what we've today is a kludge, and if we want to provide
a "seamless" user experience we need to fix it.

Don't get me wrong usbfs is a really nice solution for driving
usb devices from userspace, like scanners and all other sorts of
devices. But what all these devices have in common is that they
have no kernel driver. Having a userspace based driver and a kernel
driver "fight it out" just does not work well.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Tue, 9 Aug 2011, Hans de Goede wrote:

> Hi,
> 
> On 08/09/2011 04:19 PM, Alan Stern wrote:
> > On Tue, 9 Aug 2011, Hans de Goede wrote:
> > 
> > > I would really like to see the dual mode camera and TV tuner discussion
> > > separated. They are 2 different issues AFAIK.
> > > 
> > > 1) Dual mode cameras:
> > > 
> > > In the case of the dual mode camera we have 1 single device (both at
> > > the hardware level and at the logical block level), which can do 2 things,
> > > but not at the same time. It can stream live video data from a sensor,
> > > or it can retrieve earlier taken pictures from some picture memory.
> > > 
> > > Unfortunately even though these 2 functions live in a single logical
> > > block,
> > > historically we've developed 2 drivers for them. This leads to fighting
> > > over device ownership (surprise surprise), and to me the solution is
> > > very clear, 1 logical block == 1 driver.
> > 
> > According to Theodore, we have developed 5 drivers for them because the
> > stillcam modes in different devices use four different vendor-specific
> > drivers.
> 
> Yes, but so the the webcam modes of the different devices, so for
> the 5 (not sure if that is the right number) dual-cam mode chipsets
> we support there will be 5 drivers, each supporting both the
> webcam and the access to pictures stored in memory of the chipset
> they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
> and both functions of the single logical device that chipset
> represents.
> 
> >  Does it really make sense to combine 5 drivers into one?
> 
> Right, that is not the plan. The plan is to simply stop having 2 drivers
> for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> + 5 webcam, to just 5 drivers. We will also likely be able to share
> code between the code for the 2 functionalities for things like generic
> set / get register functions, initialization, etc.

Probably this is a good time for some inventory to be done.

 sure of the exact number of drivers. The jeilinj driver does not count 
here. It is a standard mass storage device as a still camera, and if it is 
going to be hooked up as a webcam then it has to have some buttons pushed 
just so, prior to hookup -- after which it comes up with a different USB 
Product number as a proprietary webcam device, supported by the jeilinj 
driver. Thus, this is not one of the droids we are looking for.

Specifically, the currently affected drivers are sq905, sq905c, mr97310a, 
and sn9c2028. 

Fuji Finepix seems to produce both still cameras (PTP protocol) and 
webcams. If there are currently any dual-mode cameras, it is not obvious. 
Perhaps any of these which can be dual mode are like the jeilinj cameras 
and come up under different Product numbers in the different modes, which 
is not a problem. 

There do seem to exist stv06xx dual-mode cameras:

stv0680/stv0680.c:      { "STM:USB Dual-mode camera",   0x0553, 0x0202, 1 
},

is found in camlibs/stv0680, for example, along with a whole bunch of 
other cameras with the same Vendor:Product number. It seems, too, that the 
same Vendor:Product number is listed in gspca/stv0680.c

So I guess we found another affected driver.


The intersection seems to be void between the lists of the spca50x cameras 
and webcams, though one can not be certain about the future. I do know of 
another dualmode camera, powered by the JL2005A chip. I wrote a kernel 
module for it some months ago. Said module has not been sent upward 
because of mysterious issues with one of the three cameras used for 
testing it.

We also find among the gspca driver files this

* GSPCA sub driver for W996[78]CF JPEG USB Dual Mode Camera Chip.
 *
 * Copyright (C) 2009 Hans de Goede <hdegoede@redhat.com>

but I am not sure about which cameras are using this.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Xiaofan Chen]

On Wed, Aug 10, 2011 at 4:57 AM, Hans de Goede <hdegoede@redhat.com> wrote:
> This has been discussed over and over and over again, playing clever
> tricks with USBDEVFS_[DIS]CONNECT like adding a new
> USBDEVFS_TRYDISCONNECT
> which the v4l2 driver could intercept won't cut it. We need some central
> manager of the device doing multiplexing between the 2 functions, and you
> can *not* assume that either side will be nice wrt closing file descriptors.

So it seems to be both good and bad for Linux's ability to do
multiplexing between the two drivers (one purpose-built kernel driver
and one generic usbfs driver to be used libusb). The good thing
is that it is quite useful in many use cases. The ability to detach
the existing kernel driver and use usbfs is the base for many libusb
based program under Linux and it beats Windows and Mac OS X
handsomely in this aspect. On the other hand, based on the
discussions, it seems to adds quite some issues/complexities
in some other use cases.

Windows typically does not have such an issue. You can use
only one driver. In order to use libusb-win32 or libusb-1.0 Windows
backend, you have to replace the existing kernel driver with
libusb-win32 kernel driver (libusb0.sys) or libusb-1.0 Windows
backend driver (currently winusb.sys).

The exception is the libusb0.sys filter driver. In that case, you
use libusb0.sys as the upper filter driver on top of the existing
kernel driver. On the other hand, libusb-win32 filter driver is not
really widely used due to issues with older versions.

> All in all what we've today is a kludge, and if we want to provide
> a "seamless" user experience we need to fix it.
>
> Don't get me wrong usbfs is a really nice solution for driving
> usb devices from userspace, like scanners and all other sorts of
> devices. But what all these devices have in common is that they
> have no kernel driver. Having a userspace based driver and a kernel
> driver "fight it out" just does not work well.
>

All in all, I think this is a good summary. There are cases where
a kernel driver and a usbfs are both used, an example is FTDI
device where ftdi-sio driver and the user space libftdi (based
on libusb) are used, but there are problems with that as well
with regard to switching between the two drivers.

There are a few discussions here.
http://libusb.6.n5.nabble.com/Patch-libusb-os-linux-usbfs-c-Distingush-between-usbfs-and-other-kernel-mode-drivers-td3199947.html
http://libusb.6.n5.nabble.com/open-device-exclusively-td4524397.html
And there are no good solutions to a "simple" issue as the
above mentioned.

Not so sure if this is a good suggestion or not: just wondering how
the other side (eg Windows) deal with these dual mode cameras?
Does Windows use single driver or does Windows use two drivers
for these dual mode cameras? Since they are customized device,
then there must be a vendor driver for them. And since they will
function as a video cam and a still cam device, I assume there
will be standard Windows drivers on top of the vendor drivers. So
there will be more than one drivers associated with them.


-- 
Xiaofan

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Tue, 9 Aug 2011, Hans de Goede wrote:

> Hi,
> 
> On 08/09/2011 04:19 PM, Alan Stern wrote:
> > On Tue, 9 Aug 2011, Hans de Goede wrote:
> > According to Theodore, we have developed 5 drivers for them because the
> > stillcam modes in different devices use four different vendor-specific
> > drivers.
> 
> Yes, but so the the webcam modes of the different devices, so for
> the 5 (not sure if that is the right number) dual-cam mode chipsets
> we support there will be 5 drivers, each supporting both the
> webcam and the access to pictures stored in memory of the chipset
> they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
> and both functions of the single logical device that chipset
> represents.
> 
> >  Does it really make sense to combine 5 drivers into one?
> 
> Right, that is not the plan. The plan is to simply stop having 2 drivers
> for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> + 5 webcam, to just 5 drivers. We will also likely be able to share
> code between the code for the 2 functionalities for things like generic
> set / get register functions, initialization, etc.

Okay, I didn't realize that the different cameras used different webcam 
drivers as well as different stillcam drivers.

As far as I can see, there's nothing to stop anybody from adding the 
stillcam functionality into the webcam drivers right now.  If some 
common code can be abstracted out into a shared source file, so much 
the better.

That would solve the problem, right?

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Alan Stern wrote:

> On Tue, 9 Aug 2011, Hans de Goede wrote:
> 
> > Hi,
> > 
> > On 08/09/2011 04:19 PM, Alan Stern wrote:
> > > On Tue, 9 Aug 2011, Hans de Goede wrote:
> > > According to Theodore, we have developed 5 drivers for them because the
> > > stillcam modes in different devices use four different vendor-specific
> > > drivers.
> > 
> > Yes, but so the the webcam modes of the different devices, so for
> > the 5 (not sure if that is the right number) dual-cam mode chipsets
> > we support there will be 5 drivers, each supporting both the
> > webcam and the access to pictures stored in memory of the chipset
> > they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
> > and both functions of the single logical device that chipset
> > represents.
> > 
> > >  Does it really make sense to combine 5 drivers into one?
> > 
> > Right, that is not the plan. The plan is to simply stop having 2 drivers
> > for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> > + 5 webcam, to just 5 drivers. We will also likely be able to share
> > code between the code for the 2 functionalities for things like generic
> > set / get register functions, initialization, etc.
> 
> Okay, I didn't realize that the different cameras used different webcam 
> drivers as well as different stillcam drivers.

Oh, yes. They are Proprietary devices. And that means what it says. :-)
And all different from each other, too.
 
> As far as I can see, there's nothing to stop anybody from adding the 
> stillcam functionality into the webcam drivers right now.  If some 
> common code can be abstracted out into a shared source file, so much 
> the better.
> 
> That would solve the problem, right?

I think everyone involved believes that it would solve the problem. 

The question has been all along whether or not there is any other way 
which would work. Also the question of what, exactly, "belongs" in the 
kernel and what does not. For, if something has been historically 
supported in userspace (stillcam support, in this case) and has worked 
well there, I would think it is kind of too bad to have to move said 
support into the kernel just because the same hardware requires kernel 
support for another functionality and the two sides clash. I mean, the 
kernel is already big enough, no? But the logic that Hans has set forth 
seems rather compelling. 

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Wed, 10 Aug 2011, Theodore Kilgore wrote:

> > Okay, I didn't realize that the different cameras used different webcam 
> > drivers as well as different stillcam drivers.
> 
> Oh, yes. They are Proprietary devices. And that means what it says. :-)
> And all different from each other, too.
>  
> > As far as I can see, there's nothing to stop anybody from adding the 
> > stillcam functionality into the webcam drivers right now.  If some 
> > common code can be abstracted out into a shared source file, so much 
> > the better.
> > 
> > That would solve the problem, right?
> 
> I think everyone involved believes that it would solve the problem. 
> 
> The question has been all along whether or not there is any other way 
> which would work. Also the question of what, exactly, "belongs" in the 
> kernel and what does not. For, if something has been historically 
> supported in userspace (stillcam support, in this case) and has worked 
> well there, I would think it is kind of too bad to have to move said 
> support into the kernel just because the same hardware requires kernel 
> support for another functionality and the two sides clash. I mean, the 
> kernel is already big enough, no? But the logic that Hans has set forth 
> seems rather compelling. 

The alternative seems to be to define a device-sharing protocol for USB
drivers.  Kernel drivers would implement a new callback (asking them to
give up control of the device), and usbfs would implement new ioctls by
which a program could ask for and relinquish control of a device.  The
amount of rewriting needed would be relatively small.

A few loose ends would remain, such as how to handle suspends, resumes,
resets, and disconnects.  Assuming usbfs is the only driver that will
want to share a device in this way, we could handle them.

Hans, what do you think?

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Alan Stern wrote:

> On Wed, 10 Aug 2011, Theodore Kilgore wrote:
> 
> > > Okay, I didn't realize that the different cameras used different webcam 
> > > drivers as well as different stillcam drivers.
> > 
> > Oh, yes. They are Proprietary devices. And that means what it says. :-)
> > And all different from each other, too.
> >  
> > > As far as I can see, there's nothing to stop anybody from adding the 
> > > stillcam functionality into the webcam drivers right now.  If some 
> > > common code can be abstracted out into a shared source file, so much 
> > > the better.
> > > 
> > > That would solve the problem, right?
> > 
> > I think everyone involved believes that it would solve the problem. 
> > 
> > The question has been all along whether or not there is any other way 
> > which would work. Also the question of what, exactly, "belongs" in the 
> > kernel and what does not. For, if something has been historically 
> > supported in userspace (stillcam support, in this case) and has worked 
> > well there, I would think it is kind of too bad to have to move said 
> > support into the kernel just because the same hardware requires kernel 
> > support for another functionality and the two sides clash. I mean, the 
> > kernel is already big enough, no? But the logic that Hans has set forth 
> > seems rather compelling. 
> 
> The alternative seems to be to define a device-sharing protocol for USB
> drivers.  Kernel drivers would implement a new callback (asking them to
> give up control of the device), and usbfs would implement new ioctls by
> which a program could ask for and relinquish control of a device.  The
> amount of rewriting needed would be relatively small.
> 
> A few loose ends would remain, such as how to handle suspends, resumes,
> resets, and disconnects.  Assuming usbfs is the only driver that will
> want to share a device in this way, we could handle them.
> 
> Hans, what do you think?
> 
> Alan Stern

Alan,

Hans seems to have argued cogently for doing all of this in the kernel and 
for abandoning the usbfs-based drivers for these particular drivers for 
dual-mode cameras and, I would conjecture, for drivers for dual-mode 
hardware in general. Therefore, I anticipate that he won't like that very 
much.

My position:

I do not have preconceptions about how the problem gets handled, and at 
this point I remain agnostic and believe that all approaches ought to be 
carefully analysed. I can imagine, abstractly, that things like this 
could be handled by

-- moving all basic functionality to the kernel, and fixing the 
relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
(What Hans argues for, and I am not opposed if his arguments convince 
other concerned parties)

-- doing some kind of patch job to make current arrangement somehow to 
work better (this seems to be the position of Adam Baker; I do share
the skepticism Hans has expressed about how well this could all be 
pasted together)

-- doing something like the previous, but also figuring out how to bring 
udev rules into play, which would make it all work better (just tossing 
this one in, for laughs, but who knows someone might like it)

-- moving the kernel webcam drivers out of the kernel and doing with these 
cameras _everything_ including webcam function through libusb. I myself do 
not have the imagination to be able to figure out how this could be done 
without a rather humongous amount of work (for example, which streaming 
apps that are currently available would be able to live with this?) but 
unless I misunderstand what he was saying, Greg K-H seems to think that 
this would be the best thing to do.

Which one of these possibile approaches gets adopted is a policy issue 
which I would consider is ultimately way above my pay grade.

My main motivation for bringing up the issue was to get it to the front 
burner so that _something_ gets done. It is a matter which has been left 
alone for too long. Therefore, I am very glad that the matter is being 
addressed.

Let me add to this that I have gotten permission for time off and for a 
expense money which might possibly cover my air fare. I hope to arrive in 
Vancouver by sometime on Monday and intend to attend the mini-summit. I 
suggest that we get all intersted parties together and figure out what is 
the best way to go.

I hope everyone who is actively concerned can meet in Vancouver, and if 
all goes well then on Monday as well as Tuesday. I can hang around for 
another day or two after Tuesday, but I do not expect to register for 
LinuxCon or be involved in it. When I leave Vancouver I will probably go 
to Seattle and spend a couple of days with my oldest son, the musician, 
before coming home on the next weekend.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Hans Verkuil]

On Wednesday, August 10, 2011 20:33:25 Theodore Kilgore wrote:
> 
> On Wed, 10 Aug 2011, Alan Stern wrote:
> 
> > On Wed, 10 Aug 2011, Theodore Kilgore wrote:
> > 
> > > > Okay, I didn't realize that the different cameras used different webcam 
> > > > drivers as well as different stillcam drivers.
> > > 
> > > Oh, yes. They are Proprietary devices. And that means what it says. :-)
> > > And all different from each other, too.
> > >  
> > > > As far as I can see, there's nothing to stop anybody from adding the 
> > > > stillcam functionality into the webcam drivers right now.  If some 
> > > > common code can be abstracted out into a shared source file, so much 
> > > > the better.
> > > > 
> > > > That would solve the problem, right?
> > > 
> > > I think everyone involved believes that it would solve the problem. 
> > > 
> > > The question has been all along whether or not there is any other way 
> > > which would work. Also the question of what, exactly, "belongs" in the 
> > > kernel and what does not. For, if something has been historically 
> > > supported in userspace (stillcam support, in this case) and has worked 
> > > well there, I would think it is kind of too bad to have to move said 
> > > support into the kernel just because the same hardware requires kernel 
> > > support for another functionality and the two sides clash. I mean, the 
> > > kernel is already big enough, no? But the logic that Hans has set forth 
> > > seems rather compelling. 
> > 
> > The alternative seems to be to define a device-sharing protocol for USB
> > drivers.  Kernel drivers would implement a new callback (asking them to
> > give up control of the device), and usbfs would implement new ioctls by
> > which a program could ask for and relinquish control of a device.  The
> > amount of rewriting needed would be relatively small.
> > 
> > A few loose ends would remain, such as how to handle suspends, resumes,
> > resets, and disconnects.  Assuming usbfs is the only driver that will
> > want to share a device in this way, we could handle them.
> > 
> > Hans, what do you think?
> > 
> > Alan Stern
> 
> Alan,
> 
> Hans seems to have argued cogently for doing all of this in the kernel and 
> for abandoning the usbfs-based drivers for these particular drivers for 
> dual-mode cameras and, I would conjecture, for drivers for dual-mode 
> hardware in general. Therefore, I anticipate that he won't like that very 
> much.
> 
> My position:
> 
> I do not have preconceptions about how the problem gets handled, and at 
> this point I remain agnostic and believe that all approaches ought to be 
> carefully analysed. I can imagine, abstractly, that things like this 
> could be handled by
> 
> -- moving all basic functionality to the kernel, and fixing the 
> relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
> (What Hans argues for, and I am not opposed if his arguments convince 
> other concerned parties)
> 
> -- doing some kind of patch job to make current arrangement somehow to 
> work better (this seems to be the position of Adam Baker; I do share
> the skepticism Hans has expressed about how well this could all be 
> pasted together)
> 
> -- doing something like the previous, but also figuring out how to bring 
> udev rules into play, which would make it all work better (just tossing 
> this one in, for laughs, but who knows someone might like it)
> 
> -- moving the kernel webcam drivers out of the kernel and doing with these 
> cameras _everything_ including webcam function through libusb. I myself do 
> not have the imagination to be able to figure out how this could be done 
> without a rather humongous amount of work (for example, which streaming 
> apps that are currently available would be able to live with this?) but 
> unless I misunderstand what he was saying, Greg K-H seems to think that 
> this would be the best thing to do.

Webcam drivers can rely on i2c sensor drivers to handle the sensor. These
sensor drivers can be used in other non-USB devices as well (particularly
embedded SoC devices). Moving webcam drivers out of the kernel would likely
mean that you end up with duplicate kernel and userspace sensor drivers.
Not a good idea.

In all fairness I have to say that most webcam drivers program the
sensor directly instead of relying on an i2c driver. Often because they
were reverse engineered. But I know of several that should really be split
into a webcam driver and a separate i2c sensor driver.

Video hardware tends to rely heavily on i2c devices hooked up to them. The
big advantage of keeping everything in the kernel is that that makes i2c
driver sharing between USB, PCI, SoC, etc. much easier. Video hardware
also often has to toggle between different modes or functionalities (e.g.
analog TV vs DVB, MPEG encoded video vs raw). Having all components in the
kernel makes this a much easier job.

I think the approach favored by Hans de Goede is the right one based on what
I've read.

Whatever solution is chosen, in no way should that ever force us to duplicate
i2c sensor control in userspace. This would completely defeat all our efforts
to make sensor drivers reusable between webcams and SoCs.

Regards,

	Hans

> 
> Which one of these possibile approaches gets adopted is a policy issue 
> which I would consider is ultimately way above my pay grade.
> 
> My main motivation for bringing up the issue was to get it to the front 
> burner so that _something_ gets done. It is a matter which has been left 
> alone for too long. Therefore, I am very glad that the matter is being 
> addressed.
> 
> Let me add to this that I have gotten permission for time off and for a 
> expense money which might possibly cover my air fare. I hope to arrive in 
> Vancouver by sometime on Monday and intend to attend the mini-summit. I 
> suggest that we get all intersted parties together and figure out what is 
> the best way to go.
> 
> I hope everyone who is actively concerned can meet in Vancouver, and if 
> all goes well then on Monday as well as Tuesday. I can hang around for 
> another day or two after Tuesday, but I do not expect to register for 
> LinuxCon or be involved in it. When I leave Vancouver I will probably go 
> to Seattle and spend a couple of days with my oldest son, the musician, 
> before coming home on the next weekend.
> 
> Theodore Kilgore
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Wed, Aug 10, 2011 at 01:33:25PM -0500, Theodore Kilgore wrote:
> -- moving the kernel webcam drivers out of the kernel and doing with these 
> cameras _everything_ including webcam function through libusb. I myself do 
> not have the imagination to be able to figure out how this could be done 
> without a rather humongous amount of work (for example, which streaming 
> apps that are currently available would be able to live with this?) but 
> unless I misunderstand what he was saying, Greg K-H seems to think that 
> this would be the best thing to do.

No, I never said that.  Or if I accidentally did, I do not think that is
the best thing to do at all, sorry for any confusion.

> Which one of these possibile approaches gets adopted is a policy issue 
> which I would consider is ultimately way above my pay grade.

As no one is being paid here, there are no "pay grades", so don't worry
about that :)

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 10-08-2011 15:33, Theodore Kilgore escreveu:

> Hans seems to have argued cogently for doing all of this in the kernel and 
> for abandoning the usbfs-based drivers for these particular drivers for 
> dual-mode cameras and, I would conjecture, for drivers for dual-mode 
> hardware in general. Therefore, I anticipate that he won't like that very 
> much.
> 
> My position:
> 
> I do not have preconceptions about how the problem gets handled, and at 
> this point I remain agnostic and believe that all approaches ought to be 
> carefully analysed. I can imagine, abstractly, that things like this 
> could be handled by
> 
> -- moving all basic functionality to the kernel, and fixing the 
> relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
> (What Hans argues for, and I am not opposed if his arguments convince 
> other concerned parties)

Not looking on the amount of work to be done, I think that this would
give better results, IMO.

> -- doing some kind of patch job to make current arrangement somehow to 
> work better (this seems to be the position of Adam Baker; I do share
> the skepticism Hans has expressed about how well this could all be 
> pasted together)

Adam Baker's proposal of a locking between usbfs and the kernel driver seems
to be interesting, but, as he pointed, there are some side effects to consider,
like suspend/resume, PM, etc.

> -- doing something like the previous, but also figuring out how to bring 
> udev rules into play, which would make it all work better (just tossing 
> this one in, for laughs, but who knows someone might like it)

I don't think this is a good alternative.

> -- moving the kernel webcam drivers out of the kernel and doing with these 
> cameras _everything_ including webcam function through libusb. I myself do 
> not have the imagination to be able to figure out how this could be done 
> without a rather humongous amount of work (for example, which streaming 
> apps that are currently available would be able to live with this?) but 
> unless I misunderstand what he was saying, Greg K-H seems to think that 
> this would be the best thing to do.

I also don't think that this a good alternative. As Hans V. pointed, one of
our long term targets is to create per-sensor I2C drivers that are independent
from the bridges. Also, moving it to userspace would require lots of work
with the duplication of V4L and gspca core into userspace for the devices
that would be moved, and may have some performance impacts.

> Which one of these possibile approaches gets adopted is a policy issue 
> which I would consider is ultimately way above my pay grade.
> 
> My main motivation for bringing up the issue was to get it to the front 
> burner so that _something_ gets done. It is a matter which has been left 
> alone for too long. Therefore, I am very glad that the matter is being 
> addressed.
> 
> Let me add to this that I have gotten permission for time off and for a 
> expense money which might possibly cover my air fare. I hope to arrive in 
> Vancouver by sometime on Monday and intend to attend the mini-summit. I 
> suggest that we get all intersted parties together and figure out what is 
> the best way to go.
> 
> I hope everyone who is actively concerned can meet in Vancouver, and if 
> all goes well then on Monday as well as Tuesday. I can hang around for 
> another day or two after Tuesday, but I do not expect to register for 
> LinuxCon or be involved in it.

It will be great to have you there for those discussions.

> When I leave Vancouver I will probably go 
> to Seattle and spend a couple of days with my oldest son, the musician, 
> before coming home on the next weekend.
> 
> Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Greg KH wrote:

> On Wed, Aug 10, 2011 at 01:33:25PM -0500, Theodore Kilgore wrote:
> > -- moving the kernel webcam drivers out of the kernel and doing with these 
> > cameras _everything_ including webcam function through libusb. I myself do 
> > not have the imagination to be able to figure out how this could be done 
> > without a rather humongous amount of work (for example, which streaming 
> > apps that are currently available would be able to live with this?) but 
> > unless I misunderstand what he was saying, Greg K-H seems to think that 
> > this would be the best thing to do.
> 
> No, I never said that.  Or if I accidentally did, I do not think that is
> the best thing to do at all, sorry for any confusion.

OK.
 
> > Which one of these possibile approaches gets adopted is a policy issue 
> > which I would consider is ultimately way above my pay grade.
> 
> As no one is being paid here, there are no "pay grades", so don't worry
> about that :)

I meant this, of course, in a metaphorical sense.

Cheers,
Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Mauro Carvalho Chehab wrote:

> Em 10-08-2011 15:33, Theodore Kilgore escreveu:
> 
> > Hans seems to have argued cogently for doing all of this in the kernel and 
> > for abandoning the usbfs-based drivers for these particular drivers for 
> > dual-mode cameras and, I would conjecture, for drivers for dual-mode 
> > hardware in general. Therefore, I anticipate that he won't like that very 
> > much.
> > 
> > My position:
> > 
> > I do not have preconceptions about how the problem gets handled, and at 
> > this point I remain agnostic and believe that all approaches ought to be 
> > carefully analysed. I can imagine, abstractly, that things like this 
> > could be handled by
> > 
> > -- moving all basic functionality to the kernel, and fixing the 
> > relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
> > (What Hans argues for, and I am not opposed if his arguments convince 
> > other concerned parties)
> 
> Not looking on the amount of work to be done, I think that this would
> give better results, IMO.

Okay. I would guess that I am one of the guys who gets to do the work, 
though.

> 
> > -- doing some kind of patch job to make current arrangement somehow to 
> > work better (this seems to be the position of Adam Baker; I do share
> > the skepticism Hans has expressed about how well this could all be 
> > pasted together)
> 
> Adam Baker's proposal of a locking between usbfs and the kernel driver seems
> to be interesting, but, as he pointed, there are some side effects to consider,
> like suspend/resume, PM, etc.
> 
> > -- doing something like the previous, but also figuring out how to bring 
> > udev rules into play, which would make it all work better (just tossing 
> > this one in, for laughs, but who knows someone might like it)
> 
> I don't think this is a good alternative.

I was trying to mention all alternatives. I should have also mentioned 
"leave things the way they are" but that is certainly out the window.

> 
> > -- moving the kernel webcam drivers out of the kernel and doing with these 
> > cameras _everything_ including webcam function through libusb. I myself do 
> > not have the imagination to be able to figure out how this could be done 
> > without a rather humongous amount of work (for example, which streaming 
> > apps that are currently available would be able to live with this?) but 
> > unless I misunderstand what he was saying, Greg K-H seems to think that 
> > this would be the best thing to do.
> 
> I also don't think that this a good alternative. As Hans V. pointed, one of
> our long term targets is to create per-sensor I2C drivers that are independent
> from the bridges. Also, moving it to userspace would require lots of work
> with the duplication of V4L and gspca core into userspace for the devices
> that would be moved, and may have some performance impacts.

A good argument, though it probably does not affect the devices on my list 
one way or the other. 

> 
> > Which one of these possibile approaches gets adopted is a policy issue 
> > which I would consider is ultimately way above my pay grade.
> > 
> > My main motivation for bringing up the issue was to get it to the front 
> > burner so that _something_ gets done. It is a matter which has been left 
> > alone for too long. Therefore, I am very glad that the matter is being 
> > addressed.
> > 
> > Let me add to this that I have gotten permission for time off and for a 
> > expense money which might possibly cover my air fare. I hope to arrive in 
> > Vancouver by sometime on Monday and intend to attend the mini-summit. I 
> > suggest that we get all intersted parties together and figure out what is 
> > the best way to go.
> > 
> > I hope everyone who is actively concerned can meet in Vancouver, and if 
> > all goes well then on Monday as well as Tuesday. I can hang around for 
> > another day or two after Tuesday, but I do not expect to register for 
> > LinuxCon or be involved in it.
> 
> It will be great to have you there for those discussions.

My take on this was that it seems to have become important for me to 
attend, which, frankly, I was not expecting. So thanks for the nice words.

> 
> > When I leave Vancouver I will probably go 
> > to Seattle and spend a couple of days with my oldest son, the musician, 
> > before coming home on the next weekend.
> > 
> > Theodore Kilgore
> 

Re: USB mini-summit at LinuxCon Vancouver

[Adam Baker]

On Tuesday 09 August 2011, Hans de Goede wrote:
> Hi,
> 
> On 08/09/2011 10:31 PM, Adam Baker wrote:
> > On Tuesday 09 August 2011, Hans de Goede wrote:
> <snip>
> 
> > It has also just occured to me that it might be possible to solve the
> > issues we are facing just in the kernel. At the moment when the kernel
> > performs a USBDEVFS_DISCONNECT it keeps the kernel driver locked out
> > until userspace performs a USBDEVFS_CONNECT. If the kernel reattached
> > the kernel driver when the device file was closed then, as gvfs doesn't
> > keep the file open the biggest current issue would be solved instantly.
> > If a mechanism could be found to prevent USBDEVFS_DISCONNECT from
> > succeeding when the corresponding /dev/videox file was open then that
> > would seem to be a reasonable solution.
> 
> <sigh>
> 
> This has been discussed over and over and over again, playing clever
> tricks with USBDEVFS_[DIS]CONNECT like adding a new USBDEVFS_TRYDISCONNECT
> which the v4l2 driver could intercept won't cut it. We need some central
> manager of the device doing multiplexing between the 2 functions, and you
> can *not* assume that either side will be nice wrt closing file
> descriptors.
> 
> Examples:
> 1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the
> time, and through that the usbfs device nodes.

It seems that that depends, on my system gvfs isn't actually automounting the 
camera after it detects it and the file is only open (according to lsof) when 
the device is actually mounted. As soon as you unmount it the device gets 
closed again. Because it does do a brief open,  USBDEVFS_DISCONNECT then close 
at connection time it does still disable the kernel driver.

> 
> 2) Lets say a user starts a photo managing app like f-spot, and that opens
> the device through libgphoto2 on startup, then the user switches to another
> virtual desktop and forgets all about having f-spot open. Notice that if
> the user now tries to stream he will not get a busy error, but the app
> trying to do the streaming will simply not see the camera at all (kernel
> driver unbound /dev/video# node is gone).

This does seem like a situation where your approach could potentially give a 
better user experience. I'm wondering slightly how you define busy though. For 
webcams the streamon and streamoff ioctls tell you if you are using mmap or 
userptr transfers but you don't know if when the user has finished if they 
just use read. For stillcam mode it is again hard to determine a busy 
condition other than being in the middle of transfering an individual picture.
> 
> 3) Notice that little speaker icon in your panel on your average Linux
> desktop, that keeps the mixer of the audio device open *all the time* it
> is quite easy to imagine a similar applet for v4l2 device controls (see
> for example gtk-v4l) doing the same. Or a user could simply start up a
> v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running
> minimized ...
> 

This again needs a usable concept of busy

> 4) Some laptops have a Fn + F## key which enables / disables the builtin
> webcam by turning its power on / off. Effectively plugging it into / out
> of a usb port. We would like to have an on screen notification of this one
> day like we have now for brightness and volume controls, based on udev
> events. But the current dual mode cam stuff causes udev events for
> a *new* video device being added / an existing one being removed
> each time libgphoto2 releases / takes control of the camera.
> 

Would such a system know what camera is supposed to be the internal one so it 
doesn't show the camera as turned on just because you plug in an external 
camera. If so then it won't turn on and off as an external camera changes 
modes. If not then showing on when any camera is usable and off when it isn't 
seems like sensible behaviour.

> 5) More in general, more and more software is dynamically monitoring the
> addition / removal of (usb) devices using udev, our current solution
> suggests to this software the /dev/video device is being unplugged /
> re-plugged all the time, not pretty.
> 
> 
> All in all what we've today is a kludge, and if we want to provide
> a "seamless" user experience we need to fix it.

I think in summary I'm concerned about the possibility of perfect being the 
enemy of good enough. At the moment we've got a significant usability problem 
(a web search for gvfs-gphoto2-volume-monitor turns up mostly instructions on 
how to disable it). If we come up with a solution that whilst it would be 
perfect there isn't enough effort available to implement then that is worse 
than a solution that fixes most of the problem. This is an even greater 
concern when the technically superior solution has a higher long term 
maintenance overhead (as we no longer get Win32 and OSX users helping to 
maintain the stillcam drivers).

I'm not sure if there is anything in this discussion that is relevant to the 
cameras in phones or tablets. These appear to the user as if they are dual 
mode devices but they don't have any independent storage - taking a photo is 
more like capturing a single higher than usual res frame so I suspect they 
aren't going to be an issue.

Regards

Adam

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/10/2011 06:09 PM, Alan Stern wrote:
> On Wed, 10 Aug 2011, Theodore Kilgore wrote:
>
>>> Okay, I didn't realize that the different cameras used different webcam
>>> drivers as well as different stillcam drivers.
>>
>> Oh, yes. They are Proprietary devices. And that means what it says. :-)
>> And all different from each other, too.
>>
>>> As far as I can see, there's nothing to stop anybody from adding the
>>> stillcam functionality into the webcam drivers right now.  If some
>>> common code can be abstracted out into a shared source file, so much
>>> the better.
>>>
>>> That would solve the problem, right?
>>
>> I think everyone involved believes that it would solve the problem.
>>
>> The question has been all along whether or not there is any other way
>> which would work. Also the question of what, exactly, "belongs" in the
>> kernel and what does not. For, if something has been historically
>> supported in userspace (stillcam support, in this case) and has worked
>> well there, I would think it is kind of too bad to have to move said
>> support into the kernel just because the same hardware requires kernel
>> support for another functionality and the two sides clash. I mean, the
>> kernel is already big enough, no? But the logic that Hans has set forth
>> seems rather compelling.
>
> The alternative seems to be to define a device-sharing protocol for USB
> drivers.  Kernel drivers would implement a new callback (asking them to
> give up control of the device), and usbfs would implement new ioctls by
> which a program could ask for and relinquish control of a device.  The
> amount of rewriting needed would be relatively small.
>
> A few loose ends would remain, such as how to handle suspends, resumes,
> resets, and disconnects.  Assuming usbfs is the only driver that will
> want to share a device in this way, we could handle them.
>
> Hans, what do you think?
>

First of all thanks for the constructive input!

When you say: "device-sharing protocol", do you mean 2 drivers been
attached, but only 1 being active. Or just some additional glue to make
hand-over between them work better?

I've 2 concerns with this approach:
1) Assuming we are going for the just make hand over work better solution
we will still have the whole disappear / reappear act of the /dev/video#
node, which I don't like at all.

If for example skype gets started it will say the user has no camera. If it
were to say the device is busy, the user just might make a link to some
application using the device in stillcam mode still being open.

2) The whole notion of the device being in use is rather vague when it comes
to the userspace parts of this. Simply leaving say F-Spot running, or having
a gvfs libgphoto share mounted, should not lead to not being able to use the
device in webcam mode. But currently it will.

Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if
we do that now, more broken ones will likely come along later. I estimate
98% of all cameras are not dual mode cameras, so the average stillcam
application developer will not test for this.

That leaves us with fixing the busy notion inside libgphoto2, iow, release
the device as soon as an operation has completed. This will be quite slow,
since both drivers don't know anything about each other, they will just
know there is some $random_other_driver. So they need to assume the
device state is unclean and re-init the device from scratch each time.

Where as if we have both functions in one driver, that can remember the
actual device state and only make changes if needed, so downloading +
deleting 10 photos will lead to setting it to stillcam mode once, rather
then 20 times.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/11/2011 01:04 AM, Adam Baker wrote:
> On Tuesday 09 August 2011, Hans de Goede wrote:
>> Hi,
>>
>> On 08/09/2011 10:31 PM, Adam Baker wrote:
>>> On Tuesday 09 August 2011, Hans de Goede wrote:
>> <snip>
>>
>>> It has also just occured to me that it might be possible to solve the
>>> issues we are facing just in the kernel. At the moment when the kernel
>>> performs a USBDEVFS_DISCONNECT it keeps the kernel driver locked out
>>> until userspace performs a USBDEVFS_CONNECT. If the kernel reattached
>>> the kernel driver when the device file was closed then, as gvfs doesn't
>>> keep the file open the biggest current issue would be solved instantly.
>>> If a mechanism could be found to prevent USBDEVFS_DISCONNECT from
>>> succeeding when the corresponding /dev/videox file was open then that
>>> would seem to be a reasonable solution.
>>
>> <sigh>
>>
>> This has been discussed over and over and over again, playing clever
>> tricks with USBDEVFS_[DIS]CONNECT like adding a new USBDEVFS_TRYDISCONNECT
>> which the v4l2 driver could intercept won't cut it. We need some central
>> manager of the device doing multiplexing between the 2 functions, and you
>> can *not* assume that either side will be nice wrt closing file
>> descriptors.
>>
>> Examples:
>> 1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the
>> time, and through that the usbfs device nodes.
>
> It seems that that depends, on my system gvfs isn't actually automounting the
> camera after it detects it and the file is only open (according to lsof) when
> the device is actually mounted.

Right, but on some systems it does auto-mount, and even if it does not
auto-mount and the user then just clicks on its icon in the filemanager
to look at the photo's it will mount, and it will be non obvious to the
user to umount it.

Once mounted, the /dev/video# node will be gone. So if for example skype
gets started it will say the user has no camera. If it were to say the device
is busy, the user just might make a link to the mounted gvfs share, but even
then that is a big might.

The whole disappear / reappear act of the /dev/video# node is one of my
concerns with the current solution.

> As soon as you unmount it the device gets
> closed again. Because it does do a brief open,  USBDEVFS_DISCONNECT then close
> at connection time it does still disable the kernel driver.

Then you've a bug in your libgphoto2, Fedora had a similar bug which I fixed
(and send the fix upstream). libgphoto2 now a days does make the call to
re-attach the kernel driver.

>
>>
>> 2) Lets say a user starts a photo managing app like f-spot, and that opens
>> the device through libgphoto2 on startup, then the user switches to another
>> virtual desktop and forgets all about having f-spot open. Notice that if
>> the user now tries to stream he will not get a busy error, but the app
>> trying to do the streaming will simply not see the camera at all (kernel
>> driver unbound /dev/video# node is gone).
>
> This does seem like a situation where your approach could potentially give a
> better user experience. I'm wondering slightly how you define busy though. For
> webcams the streamon and streamoff ioctls tell you if you are using mmap or
> userptr transfers but you don't know if when the user has finished if they
> just use read. For stillcam mode it is again hard to determine a busy
> condition other than being in the middle of transfering an individual picture.

Right, so my plan is to define busy for stillcam mode as not being in
the middle of some operation (be it download or delete).

>> 3) Notice that little speaker icon in your panel on your average Linux
>> desktop, that keeps the mixer of the audio device open *all the time* it
>> is quite easy to imagine a similar applet for v4l2 device controls (see
>> for example gtk-v4l) doing the same. Or a user could simply start up a
>> v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running
>> minimized ...
>>
>
> This again needs a usable concept of busy

Actually this one is easy. Most streaming drivers already handle controls
while not streaming by remembering the set values, and these cached values
usually then get set on stream start. There are multiple reasons for
this:
- usb transfers are expensive no need to send a value to the device
   if it may change before it gets actually used
- in many cases parts of the device are powered down when not streaming,
   so sending values while not streaming just drops them in a black hole


<snip>

> If we come up with a solution that whilst it would be
> perfect there isn't enough effort available to implement then that is worse
> than a solution that fixes most of the problem.

Agreed.

> This is an even greater
> concern when the technically superior solution has a higher long term
> maintenance overhead (as we no longer get Win32 and OSX users helping to
> maintain the stillcam drivers).

I've been subscribed to the libgphoto2-devel list for quite some time now,
I believe I still have to see the first win32 / OS-X user input. We do get
occasional FreeBSD input, usually in the form of "it no longer compiles".

I'm not saying that portability is not important, but contributions
from users on other platforms should not be overrated either.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Jean-Francois Moine]

On Wed, 10 Aug 2011 17:14:05 -0300
Mauro Carvalho Chehab <mchehab@infradead.org> wrote:

> > -- moving the kernel webcam drivers out of the kernel and doing with these 
> > cameras _everything_ including webcam function through libusb. I myself do 
> > not have the imagination to be able to figure out how this could be done 
> > without a rather humongous amount of work (for example, which streaming 
> > apps that are currently available would be able to live with this?) but 
> > unless I misunderstand what he was saying, Greg K-H seems to think that 
> > this would be the best thing to do.  
> 
> I also don't think that this a good alternative. As Hans V. pointed, one of
> our long term targets is to create per-sensor I2C drivers that are independent
> from the bridges. Also, moving it to userspace would require lots of work
> with the duplication of V4L and gspca core into userspace for the devices
> that would be moved, and may have some performance impacts.

If only gspca subdrivers are concerned, moving them to usespace should
be easy: only the gspca main has to be rewritten, and its best place is
in the v4l library.

-- 
Ken ar c'hentañ	|	      ** Breizh ha Linux atav! **
Jef		|		http://moinejf.free.fr/

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 11-08-2011 05:57, Jean-Francois Moine escreveu:
> On Wed, 10 Aug 2011 17:14:05 -0300
> Mauro Carvalho Chehab <mchehab@infradead.org> wrote:
> 
>>> -- moving the kernel webcam drivers out of the kernel and doing with these 
>>> cameras _everything_ including webcam function through libusb. I myself do 
>>> not have the imagination to be able to figure out how this could be done 
>>> without a rather humongous amount of work (for example, which streaming 
>>> apps that are currently available would be able to live with this?) but 
>>> unless I misunderstand what he was saying, Greg K-H seems to think that 
>>> this would be the best thing to do.  
>>
>> I also don't think that this a good alternative. As Hans V. pointed, one of
>> our long term targets is to create per-sensor I2C drivers that are independent
>> from the bridges. Also, moving it to userspace would require lots of work
>> with the duplication of V4L and gspca core into userspace for the devices
>> that would be moved, and may have some performance impacts.
> 
> If only gspca subdrivers are concerned, moving them to usespace should
> be easy: only the gspca main has to be rewritten, and its best place is
> in the v4l library.

It would end by having two userspace drivers, as the data access driver is 
at libgphoto2. So, the problem will just be moved to userspace. You'll
additionally need to deal with conflicts between kernelspace ALSA driver
and userspace video driver.

Regards,
Mauro.

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Thu, 11 Aug 2011, Hans de Goede wrote:

> > The alternative seems to be to define a device-sharing protocol for USB
> > drivers.  Kernel drivers would implement a new callback (asking them to
> > give up control of the device), and usbfs would implement new ioctls by
> > which a program could ask for and relinquish control of a device.  The
> > amount of rewriting needed would be relatively small.
> >
> > A few loose ends would remain, such as how to handle suspends, resumes,
> > resets, and disconnects.  Assuming usbfs is the only driver that will
> > want to share a device in this way, we could handle them.
> >
> > Hans, what do you think?
> >
> 
> First of all thanks for the constructive input!
> 
> When you say: "device-sharing protocol", do you mean 2 drivers been
> attached, but only 1 being active. Or just some additional glue to make
> hand-over between them work better?

I was thinking that the webcam driver would always be attached, but 
from time to time usbfs would ask to use the device.  When the webcam 
driver gives away control, it remains bound to the device but does not 
send any URBs.  If it needs to send an URB, first it has to ask usbfs 
to give control back.

> I've 2 concerns with this approach:
> 1) Assuming we are going for the just make hand over work better solution
> we will still have the whole disappear / reappear act of the /dev/video#
> node, which I don't like at all.

That will not happen any more, because the webcam driver will always be 
bound to the device.

> If for example skype gets started it will say the user has no camera. If it
> were to say the device is busy, the user just might make a link to some
> application using the device in stillcam mode still being open.
> 
> 2) The whole notion of the device being in use is rather vague when it comes
> to the userspace parts of this. Simply leaving say F-Spot running, or having
> a gvfs libgphoto share mounted, should not lead to not being able to use the
> device in webcam mode. But currently it will.

That's true -- but it's true no matter what solution we adopt.  The
various drivers (whether in the kernel or in userspace) will have to
decide for themselves when they can give up control.

> Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if
> we do that now, more broken ones will likely come along later. I estimate
> 98% of all cameras are not dual mode cameras, so the average stillcam
> application developer will not test for this.

Not all users of libgphoto2 have to be changed; only those which manage
dual-mode cameras.  Adding a few ioctls to ask for and give up control
at the appropriate times must be a lot easier than porting the entire
driver into the kernel.

> That leaves us with fixing the busy notion inside libgphoto2, iow, release
> the device as soon as an operation has completed. This will be quite slow,
> since both drivers don't know anything about each other, they will just
> know there is some $random_other_driver. So they need to assume the
> device state is unclean and re-init the device from scratch each time.

Well, a user program can assume that the kernel driver left the device
in a clean state.  The reverse isn't always true, however -- it's one
of the drawbacks of using a userspace driver.

Besides, even though drivers don't always have to re-init the device
from scratch every time, they do send all the current settings each
time they use the device.  So maybe the extra overhead is tolerable.

> Where as if we have both functions in one driver, that can remember the
> actual device state and only make changes if needed, so downloading +
> deleting 10 photos will lead to setting it to stillcam mode once, rather
> then 20 times.

Depends how the ask-for-control ioctl is implemented.  It might return
a value indicating whether or not the webcam driver took control during
the interval when the user program wasn't using the device.  If usbfs
retained control the entire time, the program should be able to assume
the device's state hasn't changed.

I'm not claiming that this is a better solution than putting everything
in the kernel.  Just that it is a workable alternative which would
involve a lot less coding.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 11-08-2011 11:56, Alan Stern escreveu:
> On Thu, 11 Aug 2011, Hans de Goede wrote:
> 
>>> The alternative seems to be to define a device-sharing protocol for USB
>>> drivers.  Kernel drivers would implement a new callback (asking them to
>>> give up control of the device), and usbfs would implement new ioctls by
>>> which a program could ask for and relinquish control of a device.  The
>>> amount of rewriting needed would be relatively small.
>>>
>>> A few loose ends would remain, such as how to handle suspends, resumes,
>>> resets, and disconnects.  Assuming usbfs is the only driver that will
>>> want to share a device in this way, we could handle them.
>>>
>>> Hans, what do you think?
>>>
>>
>> First of all thanks for the constructive input!
>>
>> When you say: "device-sharing protocol", do you mean 2 drivers been
>> attached, but only 1 being active. Or just some additional glue to make
>> hand-over between them work better?
> 
> I was thinking that the webcam driver would always be attached, but 
> from time to time usbfs would ask to use the device.  When the webcam 
> driver gives away control, it remains bound to the device but does not 
> send any URBs.  If it needs to send an URB, first it has to ask usbfs 
> to give control back.
> 
>> I've 2 concerns with this approach:
>> 1) Assuming we are going for the just make hand over work better solution
>> we will still have the whole disappear / reappear act of the /dev/video#
>> node, which I don't like at all.
> 
> That will not happen any more, because the webcam driver will always be 
> bound to the device.
> 
>> If for example skype gets started it will say the user has no camera. If it
>> were to say the device is busy, the user just might make a link to some
>> application using the device in stillcam mode still being open.
>>
>> 2) The whole notion of the device being in use is rather vague when it comes
>> to the userspace parts of this. Simply leaving say F-Spot running, or having
>> a gvfs libgphoto share mounted, should not lead to not being able to use the
>> device in webcam mode. But currently it will.
> 
> That's true -- but it's true no matter what solution we adopt.  The
> various drivers (whether in the kernel or in userspace) will have to
> decide for themselves when they can give up control.

Between two or more kernel drivers, a resource locking mechanism like the one 
you've proposed works fine, but, when the driver is on userspace, there's one
additional issue that needs to be addressed: What happens if, for example,
if a camera application using libgphoto2 crashes? The lock will be enabled, and
the V4L driver will be locked forever. Also, if the lock is made generic enough
to protect between two different userspace applications, re-starting the
camera application won't get the control back.

To avoid such risk, kernel might need to implement some ugly hacks to detect
when the application was killed, and put the device into a sane state, if this
happens.

>> Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if
>> we do that now, more broken ones will likely come along later. I estimate
>> 98% of all cameras are not dual mode cameras, so the average stillcam
>> application developer will not test for this.
> 
> Not all users of libgphoto2 have to be changed; only those which manage
> dual-mode cameras.  Adding a few ioctls to ask for and give up control
> at the appropriate times must be a lot easier than porting the entire
> driver into the kernel.

Again, applications that won't implement this control will take the lock forever.

>> That leaves us with fixing the busy notion inside libgphoto2, iow, release
>> the device as soon as an operation has completed. This will be quite slow,
>> since both drivers don't know anything about each other, they will just
>> know there is some $random_other_driver. So they need to assume the
>> device state is unclean and re-init the device from scratch each time.
> 
> Well, a user program can assume that the kernel driver left the device
> in a clean state.  The reverse isn't always true, however -- it's one
> of the drawbacks of using a userspace driver.
> 
> Besides, even though drivers don't always have to re-init the device
> from scratch every time, they do send all the current settings each
> time they use the device.  So maybe the extra overhead is tolerable.
> 
>> Where as if we have both functions in one driver, that can remember the
>> actual device state and only make changes if needed, so downloading +
>> deleting 10 photos will lead to setting it to stillcam mode once, rather
>> then 20 times.
> 
> Depends how the ask-for-control ioctl is implemented.  It might return
> a value indicating whether or not the webcam driver took control during
> the interval when the user program wasn't using the device.  If usbfs
> retained control the entire time, the program should be able to assume
> the device's state hasn't changed.
> 
> I'm not claiming that this is a better solution than putting everything
> in the kernel.  Just that it is a workable alternative which would
> involve a lot less coding.
> 
> Alan Stern
> 

Re: USB mini-summit at LinuxCon Vancouver

[Alan Cox]

> Between two or more kernel drivers, a resource locking mechanism like the one 
> you've proposed works fine, but, when the driver is on userspace, there's one
> additional issue that needs to be addressed: What happens if, for example,
> if a camera application using libgphoto2 crashes? The lock will be enabled, and
> the V4L driver will be locked forever. Also, if the lock is made generic enough
> to protect between two different userspace applications, re-starting the
> camera application won't get the control back.

Actually there are more issues than that - you've also got to worry about
a security/permission model, and that is hard to get right, especially if
you are not very careful that anything that can be retrieved which might
violate the security model (eg the last frame on the capture) has been
blanked before handover etc.

> To avoid such risk, kernel might need to implement some ugly hacks to detect
> when the application was killed, and put the device into a sane state, if this
> happens.

At which point history says it's easier to do the job right, once, in the
kernel.

> Again, applications that won't implement this control will take the lock forever.

And applications that are touching both video (even indirectly) and still
camera may get surprise deadlocks if they accidentally reference both the
still and video device even via some library or service.

> > Well, a user program can assume that the kernel driver left the device
> > in a clean state.  The reverse isn't always true, however -- it's one

Not it cannot - the user program doesn't know

a) if the kernel driver has ever been loaded
b) the values the kernel driver leaves set (and those will change no
doubt at times)
c) if the camera has been plugged and unplugged and not yet had the
kernel driver loaded

To me it sounds like a recipe for disaster. For those tiny number of
devices involved just use V4L and if need be some small V4L tweaks to
handle still mode. In most cases the interface is basically identical and
I'd bet much of the code is identical too.

Alan

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

Alan Cox raised a bunch of good points.  I'm not going to respond to 
them; they pretty much speak for themselves.

On Thu, 11 Aug 2011, Mauro Carvalho Chehab wrote:

> Between two or more kernel drivers, a resource locking mechanism like the one 
> you've proposed works fine,

It's not a locking mechanism.  More like cooperative multi-access.

>  but, when the driver is on userspace, there's one
> additional issue that needs to be addressed: What happens if, for example,
> if a camera application using libgphoto2 crashes? The lock will be enabled, and
> the V4L driver will be locked forever. Also, if the lock is made generic enough
> to protect between two different userspace applications, re-starting the
> camera application won't get the control back.

You're wrong, because what I proposed isn't a lock.  If the user 
program dies, the usbfs device file will automatically be closed and 
then usbfs will freely give control back to the webcam driver.

If the program hangs at the wrong time, then the webcam driver won't be 
able to regain control.  At least the user will have the option of 
killing the program when this happens; a similar hang in a kernel 
driver tends to be much uglier.

Two different user programs trying to drive the device at the same time 
doesn't necessarily have to cause a problem.  At any particular moment, 
only one of them would be in control of the device.

> To avoid such risk, kernel might need to implement some ugly hacks to detect
> when the application was killed, and put the device into a sane state, if this
> happens.

No ugly hack is needed.  usbfs can directly inform the webcam driver 
whether or not the device file was closed while the user program 
retained control.  If that didn't happen, it's safe to assume that the 
program gave up control voluntarily.

> > Not all users of libgphoto2 have to be changed; only those which manage
> > dual-mode cameras.  Adding a few ioctls to ask for and give up control
> > at the appropriate times must be a lot easier than porting the entire
> > driver into the kernel.
> 
> Again, applications that won't implement this control will take the lock forever.

No, because there is no lock.  Programs that haven't been changed will 
continue to behave as they do today -- they will unbind the webcam 
driver and assume full control of the device.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Thu, 11 Aug 2011, Alan Cox wrote:

> Actually there are more issues than that - you've also got to worry about
> a security/permission model, and that is hard to get right, especially if
> you are not very careful that anything that can be retrieved which might
> violate the security model (eg the last frame on the capture) has been
> blanked before handover etc.

As far as I can tell, these same security issues exist today.  I don't 
see them getting any worse than they are now.

> And applications that are touching both video (even indirectly) and still
> camera may get surprise deadlocks if they accidentally reference both the
> still and video device even via some library or service.

No, not deadlocks.  Just -EBUSY errors.

> > > Well, a user program can assume that the kernel driver left the device
> > > in a clean state.  The reverse isn't always true, however -- it's one
> 
> Not it cannot - the user program doesn't know
> 
> a) if the kernel driver has ever been loaded
> b) the values the kernel driver leaves set (and those will change no
> doubt at times)
> c) if the camera has been plugged and unplugged and not yet had the
> kernel driver loaded

That's true.  The program can't assume that a kernel driver was ever 
bound to the device; all it can assume is that _if_ a kernel driver was 
bound then it left the device in a sane state -- whatever "sane" might 
mean in this context.

> To me it sounds like a recipe for disaster. For those tiny number of
> devices involved just use V4L and if need be some small V4L tweaks to
> handle still mode. In most cases the interface is basically identical and
> I'd bet much of the code is identical too.

I'm not against moving the whole thing into the kernel.  I'm just
pointing out that an easier-to-code-up solution will accomplish much
the same result.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Thu, 11 Aug 2011, Alan Stern wrote:

> On Thu, 11 Aug 2011, Alan Cox wrote:
> 
> > Actually there are more issues than that - you've also got to worry about
> > a security/permission model, and that is hard to get right, especially if
> > you are not very careful that anything that can be retrieved which might
> > violate the security model (eg the last frame on the capture) has been
> > blanked before handover etc.
> 
> As far as I can tell, these same security issues exist today.  I don't 
> see them getting any worse than they are now.
> 
> > And applications that are touching both video (even indirectly) and still
> > camera may get surprise deadlocks if they accidentally reference both the
> > still and video device even via some library or service.
> 
> No, not deadlocks.  Just -EBUSY errors.
> 
> > > > Well, a user program can assume that the kernel driver left the device
> > > > in a clean state.  The reverse isn't always true, however -- it's one
> > 
> > Not it cannot - the user program doesn't know
> > 
> > a) if the kernel driver has ever been loaded
> > b) the values the kernel driver leaves set (and those will change no
> > doubt at times)
> > c) if the camera has been plugged and unplugged and not yet had the
> > kernel driver loaded
> 
> That's true.  The program can't assume that a kernel driver was ever 
> bound to the device; all it can assume is that _if_ a kernel driver was 
> bound then it left the device in a sane state -- whatever "sane" might 
> mean in this context.
> 
> > To me it sounds like a recipe for disaster. For those tiny number of
> > devices involved just use V4L and if need be some small V4L tweaks to
> > handle still mode. In most cases the interface is basically identical and
> > I'd bet much of the code is identical too.
> 
> I'm not against moving the whole thing into the kernel.  I'm just
> pointing out that an easier-to-code-up solution will accomplish much
> the same result.
> 
> Alan Stern
> 

Alan,

As I said, I am agnostic, though leaning in the direction that Hans de 
Goede is pointing. What he says about a single control mechanism seems to 
make a lot of sense. If you can come up with an outline of the "easier to 
code" solution, that would be interesting, though.

I assume you are also going to be in Vancouver? If you will be there on 
Monday, then Hans and I are already planning to meet and discuss. 

BTW, as to using V4L with "tweaks" to handle still mode, it would probably 
be more difficult than is imagined. For, though the operations required to 
process still images and webcam frames are in principle similar, the 
priorities and constraints are too different. Therefore, my understanding 
is that the libgphoto2 image processing routines, not the libv4l image 
processing routines, would still be used for still images.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 11-08-2011 17:01, Theodore Kilgore escreveu:

> As I said, I am agnostic, though leaning in the direction that Hans de 
> Goede is pointing. What he says about a single control mechanism seems to 
> make a lot of sense. If you can come up with an outline of the "easier to 
> code" solution, that would be interesting, though.
> 
> I assume you are also going to be in Vancouver? If you will be there on 
> Monday, then Hans and I are already planning to meet and discuss. 
> 
> BTW, as to using V4L with "tweaks" to handle still mode, it would probably 
> be more difficult than is imagined. For, though the operations required to 
> process still images and webcam frames are in principle similar, the 
> priorities and constraints are too different. Therefore, my understanding 
> is that the libgphoto2 image processing routines, not the libv4l image 
> processing routines, would still be used for still images.

I agree with Alan Cox: most of the code that the driver needs is already there: 
register read/write routines, bulk transfer support, etc. The amount of extra 
code for adding still cam functionality is probably not big.

>From the kernel driver's perspective, it doesn't matter if the access will come
via libv4l, libgphoto2 or whatever. The driver should be able to allow simultaneous
open, while protecting the data access when userspace requests data stream or
still image retrieve.

instead of using the V4L2 device node to access the stored images, it probably makes 
more sense to use a separate device for that, that will handle a separate set of 
ioctl's, and just use read() to retrieve the image data, after selecting the desired
image number, via ioctl().

It probably makes sense to add a new set of callbacks at the gspca core in order
to handle the new device node, and letting it to avoid start streaming while the
store access is happening, and vice-versa. Alternatively, we may create a separate
"still cam" core library to handle the new device node,.

If all agree around such solution, I suggest to take the most complex case and try
to map it into the driver and core, and see how it behaves, testing with some simple
command line applications, only changing the libgphoto2 code after those initial
tests. Writing a simple code for reading still images should be easy, and we have 
already some testing tools for V4L2.

After coding the core changes that are common to all drives, I suspect that adding 
the remaining 4 drivers will be quick.

With regards to libgphoto2, all it needs to do is to test if the new device nodes
exist. If they exist, then the new code will be used. Otherwise, it will fallback
to the libusb. This way, we can incrementally add the Dual mode drivers into the
kernel.

There is one advantage on using this strategy: if, in the future, new Dual Cams
arise, one can write first a still cam kernel driver, adding V4L support later.

Cheers,
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Thu, 11 Aug 2011, Mauro Carvalho Chehab wrote:

> Em 11-08-2011 17:01, Theodore Kilgore escreveu:
> 
> > As I said, I am agnostic, though leaning in the direction that Hans de 
> > Goede is pointing. What he says about a single control mechanism seems to 
> > make a lot of sense. If you can come up with an outline of the "easier to 
> > code" solution, that would be interesting, though.
> > 
> > I assume you are also going to be in Vancouver? If you will be there on 
> > Monday, then Hans and I are already planning to meet and discuss. 
> > 
> > BTW, as to using V4L with "tweaks" to handle still mode, it would probably 
> > be more difficult than is imagined. For, though the operations required to 
> > process still images and webcam frames are in principle similar, the 
> > priorities and constraints are too different. Therefore, my understanding 
> > is that the libgphoto2 image processing routines, not the libv4l image 
> > processing routines, would still be used for still images.
> 
> I agree with Alan Cox: most of the code that the driver needs is already there: 
> register read/write routines, bulk transfer support, etc. The amount of extra 
> code for adding still cam functionality is probably not big.
> 
> >From the kernel driver's perspective, it doesn't matter if the access will come
> via libv4l, libgphoto2 or whatever. The driver should be able to allow simultaneous
> open, while protecting the data access when userspace requests data stream or
> still image retrieve.
> 
> instead of using the V4L2 device node to access the stored images, it probably makes 
> more sense to use a separate device for that, that will handle a separate set of 
> ioctl's, and just use read() to retrieve the image data, after selecting the desired
> image number, via ioctl().
> 
> It probably makes sense to add a new set of callbacks at the gspca core in order
> to handle the new device node, and letting it to avoid start streaming while the
> store access is happening, and vice-versa. Alternatively, we may create a separate
> "still cam" core library to handle the new device node,.
> 
> If all agree around such solution, I suggest to take the most complex case and try
> to map it into the driver and core, and see how it behaves, testing with some simple
> command line applications, only changing the libgphoto2 code after those initial
> tests. Writing a simple code for reading still images should be easy, and we have 
> already some testing tools for V4L2.
> 
> After coding the core changes that are common to all drives, I suspect that adding 
> the remaining 4 drivers will be quick.
> 
> With regards to libgphoto2, all it needs to do is to test if the new device nodes
> exist. If they exist, then the new code will be used. Otherwise, it will fallback
> to the libusb. This way, we can incrementally add the Dual mode drivers into the
> kernel.
> 
> There is one advantage on using this strategy: if, in the future, new Dual Cams
> arise, one can write first a still cam kernel driver, adding V4L support later.
> 
> Cheers,
> Mauro
> 

Try it out with one camera seems to me to be a reasonable way 
forward. 

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Thu, 11 Aug 2011, Theodore Kilgore wrote:

> Alan,
> 
> As I said, I am agnostic, though leaning in the direction that Hans de 
> Goede is pointing. What he says about a single control mechanism seems to 
> make a lot of sense. If you can come up with an outline of the "easier to 
> code" solution, that would be interesting, though.

That approach has been outlined in the other emails I have posted 
recently.

> I assume you are also going to be in Vancouver? If you will be there on 
> Monday, then Hans and I are already planning to meet and discuss. 

No, I'm not going to Vancouver.  However I will attend the Linux 
Plumbers conference in Santa Rosa.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/11/2011 10:32 PM, Mauro Carvalho Chehab wrote:

<snip stuff I agree with>

> instead of using the V4L2 device node to access the stored images, it probably makes
> more sense to use a separate device for that, that will handle a separate set of
> ioctl's, and just use read() to retrieve the image data, after selecting the desired
> image number, via ioctl().

I don't see a lot of added value in doing things this way. We can simply
have a set of new ioctls on the /dev/video# node for this and a new
V4L2_CAP_STILL_IMAGE to indicate the availability of these ioctls. This will keep
the driver a lot simpler then doing 2 separate device nodes for 1 device.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/11/2011 04:56 PM, Alan Stern wrote:
> On Thu, 11 Aug 2011, Hans de Goede wrote:
>
>>> The alternative seems to be to define a device-sharing protocol for USB
>>> drivers.  Kernel drivers would implement a new callback (asking them to
>>> give up control of the device), and usbfs would implement new ioctls by
>>> which a program could ask for and relinquish control of a device.  The
>>> amount of rewriting needed would be relatively small.
>>>
>>> A few loose ends would remain, such as how to handle suspends, resumes,
>>> resets, and disconnects.  Assuming usbfs is the only driver that will
>>> want to share a device in this way, we could handle them.
>>>
>>> Hans, what do you think?
>>>
>>
>> First of all thanks for the constructive input!
>>
>> When you say: "device-sharing protocol", do you mean 2 drivers been
>> attached, but only 1 being active. Or just some additional glue to make
>> hand-over between them work better?
>
> I was thinking that the webcam driver would always be attached, but
> from time to time usbfs would ask to use the device.  When the webcam
> driver gives away control, it remains bound to the device but does not
> send any URBs.  If it needs to send an URB, first it has to ask usbfs
> to give control back.
>

Oh, interesting...

<snip lots of good stuff>

> I'm not claiming that this is a better solution than putting everything
> in the kernel.  Just that it is a workable alternative which would
> involve a lot less coding.

This is definitely an interesting proposal, something to think about ...

I have 2 concerns wrt this approach:

1) It feels less clean then just having a single driver; and
2) I agree it will be less coding, but I doubt it will really be that much
less work. It will likely need less new code (but a lot can be more or
less copy pasted), but it will need changes across a wider array of
subsystems / userspace components, requiring a lot of coordinating,
getting patches merged in different projects, etc. So in the end I
think it too will be quite a bit of work.

I guess that what I'm trying to say here is, that if we are going to
spend a significant amount of time on this, we might just as well
go for the best solution we can come up with even if that is some
more work.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Alan Cox]

> instead of using the V4L2 device node to access the stored images, it probably makes 
> more sense to use a separate device for that, that will handle a separate set of 
> ioctl's, and just use read() to retrieve the image data, after selecting the desired
> image number, via ioctl().

How will you handle the permission model, what about devices where you
need the V4L2 controls to take the picture in the first place ?

It seems it should really be the same device and just a minor tweak to
the API. V4L2 already provides frame reading and mapping functionality
and the media layer provides post processing services which I could
conceive some setups needing to use.

gphoto would just open the v4l device ask for still image stuff and
discover it wasn't available.

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Fri, 12 Aug 2011, Hans de Goede wrote:

> > I'm not claiming that this is a better solution than putting everything
> > in the kernel.  Just that it is a workable alternative which would
> > involve a lot less coding.
> 
> This is definitely an interesting proposal, something to think about ...
> 
> I have 2 concerns wrt this approach:
> 
> 1) It feels less clean then just having a single driver; and

Agreed.

> 2) I agree it will be less coding, but I doubt it will really be that much
> less work. It will likely need less new code (but a lot can be more or
> less copy pasted), but it will need changes across a wider array of
> subsystems / userspace components, requiring a lot of coordinating,
> getting patches merged in different projects, etc. So in the end I
> think it too will be quite a bit of work.
> 
> I guess that what I'm trying to say here is, that if we are going to
> spend a significant amount of time on this, we might just as well
> go for the best solution we can come up with even if that is some
> more work.

Okay, go ahead.  I have no objection.

Alan Stern