USBIP protocol

USBIP protocol

[Matthew Wilcox]

I'm in the middle of implementing a userspace client for usbip and I
strongly feel that the protocol needs to be changed before it is merged.

 - I'm unconvinced that TCP is the correct protocol to be running this over.
   I understand the reluctance to use UDP, but the protocol is fundamentally
   packet-based.  If TCP is used, the delimitation of packets within the
   stream needs to be much more robust.  I've managed to wedge the VHCI driver
   a number of times in ways that just wouldn't be possible if we were using
   a packet protocol instead of a stream protocol.
 - Endianness.  This is a mess.  The usbip protocol is big-endian, but the
   encapsulated usb protocol is little-endian.  This doesn't matter to the
   people who are just tunnelling usb from one computer to another, but for
   someone implementing a usbip client, it's very confusing.
 - The protocol needs an officially assigned port number.  Port 3240 is 
   already assigned to Tony Matthews <tmatthews&triomotion.com> February
   2002 (see http://www.iana.org/assignments/port-numbers)
 - There are actually two completely different protocols in use.  First,
   the usbipd daemon listens on port 3240, and handles device discovery.
   When usbip successfully attaches to usbipd, both sides of the connection
   pass the socket fd into the kernel and the protocol changes.
 - The protocol sends a 48-byte packet header for every command (and every
   response).  It's cunningly hidden as a union.
   
I think the protocol would be immeasurably improved by going through the
IETF RFC process and getting feedback from networking experts.  Failing
that, I have some suggestions about how to improve it.  I was hoping to
get my client finished before I started mucking with the protocol though.

(I have some other comments on the implementation, but they're a separate
issue).

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Andi Kleen]

Matthew Wilcox <matthew@wil.cx> writes:

> I'm in the middle of implementing a userspace client for usbip and I
> strongly feel that the protocol needs to be changed before it is merged.
>
>  - I'm unconvinced that TCP is the correct protocol to be running this over.
>    I understand the reluctance to use UDP, but the protocol is fundamentally
>    packet-based.  If TCP is used, the delimitation of packets within the
>    stream needs to be much more robust. 

If you want reliable transport with record boundaries an alternative
would be also SCTP. Main drawback is that firewalls often don't support it
though (but presumably that wouldn't be a big issue for this)

-Andi
-- 
ak@linux.intel.com

Re: USBIP protocol

[Greg KH]

On Fri, Aug 29, 2008 at 08:02:24AM -0600, Matthew Wilcox wrote:
> 
> I'm in the middle of implementing a userspace client for usbip and I
> strongly feel that the protocol needs to be changed before it is merged.
> 
>  - I'm unconvinced that TCP is the correct protocol to be running this over.
>    I understand the reluctance to use UDP, but the protocol is fundamentally
>    packet-based.  If TCP is used, the delimitation of packets within the
>    stream needs to be much more robust.  I've managed to wedge the VHCI driver
>    a number of times in ways that just wouldn't be possible if we were using
>    a packet protocol instead of a stream protocol.

USB is fundamentally packet-based, so it kind of fits very well.

>  - Endianness.  This is a mess.  The usbip protocol is big-endian, but the
>    encapsulated usb protocol is little-endian.  This doesn't matter to the
>    people who are just tunnelling usb from one computer to another, but for
>    someone implementing a usbip client, it's very confusing.

Then just document it, no big deal.
Yeah, the current code isn't the cleanest here (sparse throws up some
warnings), but it's not that much work to fix it up, it's on my todo
list.

>  - The protocol needs an officially assigned port number.  Port 3240 is 
>    already assigned to Tony Matthews <tmatthews&triomotion.com> February
>    2002 (see http://www.iana.org/assignments/port-numbers)

Great, let's get a real number.

>  - There are actually two completely different protocols in use.  First,
>    the usbipd daemon listens on port 3240, and handles device discovery.
>    When usbip successfully attaches to usbipd, both sides of the connection
>    pass the socket fd into the kernel and the protocol changes.
>  - The protocol sends a 48-byte packet header for every command (and every
>    response).  It's cunningly hidden as a union.

Is that a real problem?

> I think the protocol would be immeasurably improved by going through the
> IETF RFC process and getting feedback from networking experts.  Failing
> that, I have some suggestions about how to improve it.  I was hoping to
> get my client finished before I started mucking with the protocol though.

Why mess with the RFC process, is that really necessary for something
like this?

Windows has had this for years, no need for a RFC there, and if we just
document this well, no need for one here either.

thanks,

greg k-h

Re: USBIP protocol

[Matthew Wilcox]

On Fri, Aug 29, 2008 at 07:30:17AM -0700, Greg KH wrote:
> On Fri, Aug 29, 2008 at 08:02:24AM -0600, Matthew Wilcox wrote:
> > 
> > I'm in the middle of implementing a userspace client for usbip and I
> > strongly feel that the protocol needs to be changed before it is merged.
> > 
> >  - I'm unconvinced that TCP is the correct protocol to be running this over.
> >    I understand the reluctance to use UDP, but the protocol is fundamentally
> >    packet-based.  If TCP is used, the delimitation of packets within the
> >    stream needs to be much more robust.  I've managed to wedge the VHCI driver
> >    a number of times in ways that just wouldn't be possible if we were using
> >    a packet protocol instead of a stream protocol.
> 
> USB is fundamentally packet-based, so it kind of fits very well.

Erm, did you not read what I wrote?  USB is packet based.  TCP isn't.
We shouldn't be using TCP here.

> >  - Endianness.  This is a mess.  The usbip protocol is big-endian, but the
> >    encapsulated usb protocol is little-endian.  This doesn't matter to the
> >    people who are just tunnelling usb from one computer to another, but for
> >    someone implementing a usbip client, it's very confusing.
> 
> Then just document it, no big deal.
> Yeah, the current code isn't the cleanest here (sparse throws up some
> warnings), but it's not that much work to fix it up, it's on my todo
> list.

I'm not talking about the code.  I'm talking about the protocol.  It's a
mess to have two different endiannesses within the same packet.

> >  - There are actually two completely different protocols in use.  First,
> >    the usbipd daemon listens on port 3240, and handles device discovery.
> >    When usbip successfully attaches to usbipd, both sides of the connection
> >    pass the socket fd into the kernel and the protocol changes.
> >  - The protocol sends a 48-byte packet header for every command (and every
> >    response).  It's cunningly hidden as a union.
> 
> Is that a real problem?

Yes, it really is.  It complicates the protocol, complicates the
implementation, introduces unnecessary state, and makes it impossible to
renegotiate on the same connection.

> > I think the protocol would be immeasurably improved by going through the
> > IETF RFC process and getting feedback from networking experts.  Failing
> > that, I have some suggestions about how to improve it.  I was hoping to
> > get my client finished before I started mucking with the protocol though.
> 
> Why mess with the RFC process, is that really necessary for something
> like this?

It helps clarify the odd corners of any protocol.  I don't have the
impression that it's a terribly heavy-weight process -- though we can
ask the netlink guys how it went for them.

> Windows has had this for years, no need for a RFC there, and if we just
> document this well, no need for one here either.

Yes, and as a result we can't interoperate with Windows.

By the way, is this actually built into Windows or just available as
several mutually incompatible and pay-for products?  I did some
searching a few months ago and didn't come up with anything official
from Microsoft.

Even if we don't go through the RFC process, just writing down the
on-wire protocol should be mandatory for taking this kind of thing into
the kernel.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Greg KH]

On Fri, Aug 29, 2008 at 08:43:54AM -0600, Matthew Wilcox wrote:
> On Fri, Aug 29, 2008 at 07:30:17AM -0700, Greg KH wrote:
> > On Fri, Aug 29, 2008 at 08:02:24AM -0600, Matthew Wilcox wrote:
> > > 
> > > I'm in the middle of implementing a userspace client for usbip and I
> > > strongly feel that the protocol needs to be changed before it is merged.
> > > 
> > >  - I'm unconvinced that TCP is the correct protocol to be running this over.
> > >    I understand the reluctance to use UDP, but the protocol is fundamentally
> > >    packet-based.  If TCP is used, the delimitation of packets within the
> > >    stream needs to be much more robust.  I've managed to wedge the VHCI driver
> > >    a number of times in ways that just wouldn't be possible if we were using
> > >    a packet protocol instead of a stream protocol.
> > 
> > USB is fundamentally packet-based, so it kind of fits very well.
> 
> Erm, did you not read what I wrote?  USB is packet based.  TCP isn't.
> We shouldn't be using TCP here.

Sorry, early morning, no coffee yet...

I think in the end, we should still use TCP otherwise you just end up
reinventing it with UDP :)

> > >  - Endianness.  This is a mess.  The usbip protocol is big-endian, but the
> > >    encapsulated usb protocol is little-endian.  This doesn't matter to the
> > >    people who are just tunnelling usb from one computer to another, but for
> > >    someone implementing a usbip client, it's very confusing.
> > 
> > Then just document it, no big deal.
> > Yeah, the current code isn't the cleanest here (sparse throws up some
> > warnings), but it's not that much work to fix it up, it's on my todo
> > list.
> 
> I'm not talking about the code.  I'm talking about the protocol.  It's a
> mess to have two different endiannesses within the same packet.

Ok, switch it all to be little endian, not a bit deal.

> > >  - There are actually two completely different protocols in use.  First,
> > >    the usbipd daemon listens on port 3240, and handles device discovery.
> > >    When usbip successfully attaches to usbipd, both sides of the connection
> > >    pass the socket fd into the kernel and the protocol changes.
> > >  - The protocol sends a 48-byte packet header for every command (and every
> > >    response).  It's cunningly hidden as a union.
> > 
> > Is that a real problem?
> 
> Yes, it really is.  It complicates the protocol, complicates the
> implementation, introduces unnecessary state, and makes it impossible to
> renegotiate on the same connection.

Fair enough, patches welcome :)

> > Windows has had this for years, no need for a RFC there, and if we just
> > document this well, no need for one here either.
> 
> Yes, and as a result we can't interoperate with Windows.

That is because (see below)

> By the way, is this actually built into Windows or just available as
> several mutually incompatible and pay-for products?  I did some
> searching a few months ago and didn't come up with anything official
> from Microsoft.

There is nothing official, there are various incompatible and pay-for
products in this area.

> Even if we don't go through the RFC process, just writing down the
> on-wire protocol should be mandatory for taking this kind of thing into
> the kernel.

Why, isn't the actual implementation better than a document?  :)

thanks,

greg k-h

Re: USBIP protocol

[Matthew Wilcox]

On Fri, Aug 29, 2008 at 07:54:07AM -0700, Greg KH wrote:
> On Fri, Aug 29, 2008 at 08:43:54AM -0600, Matthew Wilcox wrote:
> > On Fri, Aug 29, 2008 at 07:30:17AM -0700, Greg KH wrote:
> > > On Fri, Aug 29, 2008 at 08:02:24AM -0600, Matthew Wilcox wrote:
> > > > 
> > > > I'm in the middle of implementing a userspace client for usbip and I
> > > > strongly feel that the protocol needs to be changed before it is merged.
> > > > 
> > > >  - I'm unconvinced that TCP is the correct protocol to be running this over.
> > > >    I understand the reluctance to use UDP, but the protocol is fundamentally
> > > >    packet-based.  If TCP is used, the delimitation of packets within the
> > > >    stream needs to be much more robust.  I've managed to wedge the VHCI driver
> > > >    a number of times in ways that just wouldn't be possible if we were using
> > > >    a packet protocol instead of a stream protocol.
> > > 
> > > USB is fundamentally packet-based, so it kind of fits very well.
> > 
> > Erm, did you not read what I wrote?  USB is packet based.  TCP isn't.
> > We shouldn't be using TCP here.
> 
> Sorry, early morning, no coffee yet...
> 
> I think in the end, we should still use TCP otherwise you just end up
> reinventing it with UDP :)

Which brings us to the alternate -- that we need better framing in the
protocol.

> Ok, switch it all to be little endian, not a bit deal.

No, but it does need agreement ;-)

> > > >  - There are actually two completely different protocols in use.  First,
> > > >    the usbipd daemon listens on port 3240, and handles device discovery.
> > > >    When usbip successfully attaches to usbipd, both sides of the connection
> > > >    pass the socket fd into the kernel and the protocol changes.
> > > >  - The protocol sends a 48-byte packet header for every command (and every
> > > >    response).  It's cunningly hidden as a union.
> > > 
> > > Is that a real problem?
> > 
> > Yes, it really is.  It complicates the protocol, complicates the
> > implementation, introduces unnecessary state, and makes it impossible to
> > renegotiate on the same connection.
> 
> Fair enough, patches welcome :)

Patches don't seem appropriate for a design discussion.  I'm more than
happy to make suggestions about how to unify the two protocols.  I'll
send a followup to this with some ideas.

> > Even if we don't go through the RFC process, just writing down the
> > on-wire protocol should be mandatory for taking this kind of thing into
> > the kernel.
> 
> Why, isn't the actual implementation better than a document?  :)

Surely you know that writing things down forces you to understand it
better?

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

RE: USBIP protocol

[Dave Higton]

Matthew Wilcox wrote:

> On Fri, Aug 29, 2008 at 07:54:07AM -0700, Greg KH wrote:
> > 
> > Why, isn't the actual implementation better than a document?  :)
> 
> Surely you know that writing things down forces you to understand it
> better?

It's an enormous help for anyone working on the code, either now or
in the future.  Reverse engineering a protocol from source code is
difficult.

Dave



*************************************************************************************************************************************************************************************************************************************************

NICE CTI Systems UK Limited ("NICE") is registered in England under company number, 3403044.  The registered office of NICE is at Tollbar Way, Hedge End, Southampton, Hampshire SO30 2ZP.

Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.

Monitoring: NICE may monitor incoming and outgoing e-mails.

Viruses:  Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.

****************************************************************************************************************************************************************************************************************************************************

 

Re: USBIP protocol

[Matthew Wilcox]

On Sat, Aug 30, 2008 at 12:31:43AM +0200, Marcel Holtmann wrote:
> Hi Matthew,
> 
> > > I'm in the middle of implementing a userspace client for usbip and I
> > > strongly feel that the protocol needs to be changed before it is merged.
> > >
> > >  - I'm unconvinced that TCP is the correct protocol to be running this over.
> > >    I understand the reluctance to use UDP, but the protocol is fundamentally
> > >    packet-based.  If TCP is used, the delimitation of packets within the
> > >    stream needs to be much more robust. 
> > 
> > If you want reliable transport with record boundaries an alternative
> > would be also SCTP. Main drawback is that firewalls often don't support it
> > though (but presumably that wouldn't be a big issue for this)
> 
> I would also have proposed SCTP. The telco carries seems be very happy
> with it.

I'm actually looking into the sunrpc protocol.  That has the advantages:
 - Already has an in-kernel implementation
 - Widely understood and properly documented
 - Can be run over TCP or UDP or even RDMA

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Willy Tarreau]

On Fri, Aug 29, 2008 at 02:46:27PM -0600, Matthew Wilcox wrote:
> On Sat, Aug 30, 2008 at 12:31:43AM +0200, Marcel Holtmann wrote:
(...)
> > I would also have proposed SCTP. The telco carries seems be very happy
> > with it.
> 
> I'm actually looking into the sunrpc protocol.  That has the advantages:
>  - Already has an in-kernel implementation
>  - Widely understood and properly documented
>  - Can be run over TCP or UDP or even RDMA

and :
   - already supported by most firewalls.

Willy

Re: USBIP protocol

[Marcel Holtmann]

Hi Matthew,

> > I'm in the middle of implementing a userspace client for usbip and I
> > strongly feel that the protocol needs to be changed before it is merged.
> >
> >  - I'm unconvinced that TCP is the correct protocol to be running this over.
> >    I understand the reluctance to use UDP, but the protocol is fundamentally
> >    packet-based.  If TCP is used, the delimitation of packets within the
> >    stream needs to be much more robust. 
> 
> If you want reliable transport with record boundaries an alternative
> would be also SCTP. Main drawback is that firewalls often don't support it
> though (but presumably that wouldn't be a big issue for this)

I would also have proposed SCTP. The telco carries seems be very happy
with it.

Regards

Marcel

Re: USBIP protocol

[Matthew Wilcox]

On Fri, Aug 29, 2008 at 07:54:07AM -0700, Greg KH wrote:
> Fair enough, patches welcome :)

Here's a document for discussion.  No code yet, though I'm quite willing
to modify the current usbip code to follow this new protocol.  Just a
matter of time.

http://www.kernel.org/pub/linux/kernel/people/willy/usbip/usbip-protocol-draft-1

To save some time for reviewers, here's a list of decisions I took while
working on the document.  I appreciate that some of the decisions I made
were not necessarily those another designer might have made, so I ask
that any comments along the lines of "I would have done it differently"
include a really good reason.

----

I've merged the two protocols.  It is no longer the case that the protocol
completely changes when userspace drops the connection into the kernel.
It's still possible to do a hybrid implementation where userspace
implements version, list and claim and kernelspace handles release,
submit and unlink.

During development of my own implementation, I noticed the client and
server get confused about where the packets were in the TCP stream.
No more; if using a stream protocol, we wrap each packet in a 'RM'.

I considered using the whole of sunrpc.  The encoding is quite
heavyweight and I felt I could do better by specialising the protocol to
USB's purposes.

'get version' is now a call rather than including a version field in
every request.

Devices are now referred to as an ascii string rather than an encoded
4-byte quantity.  This helps userspace configure the device and lets us
interoperate with other OSes that might want to implement this protocol.

Instead of transmitting the device number in every command, we now bind
each socket to a particular device.  This was already what happened,
so it was just overhead.

I've split 'cmd_submit' into four commands (control, data, isoc, irq).
That gives us the ability to make 'data' very small.

There's no need to respond with the call number to each call -- the
caller should be using the call identifier to find out what type of
call it was.  Often the implementation will issue a command and then
wait for the response, so even that is unnecessary.

Replying with the status is vital.

I decided to make the 'call' value 32-bit (instead of 8-bit) to make
everything align nicely.  Then I wanted to slim down the data command
call, so I tucked the endpoint and direction in there too.

I have no experience with isosynchronous transactions, nor interrupt
transactions, so I decline to define them at this moment.

I've given up on the big/little endian thing.  Network protocols are
traditionally BE, USB is LE and it can encapsulate SCSI which is BE again.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Alan Stern]

On Tue, 2 Sep 2008, Matthew Wilcox wrote:

> Here's a document for discussion.  No code yet, though I'm quite willing
> to modify the current usbip code to follow this new protocol.  Just a
> matter of time.
> 
> http://www.kernel.org/pub/linux/kernel/people/willy/usbip/usbip-protocol-draft-1

Most of your decisions look good to me.

> 'get version' is now a call rather than including a version field in
> every request.

Have you given any thought to forward compatibility?  I imagine it's 
too early to worry about that yet.

> Devices are now referred to as an ascii string rather than an encoded
> 4-byte quantity.  This helps userspace configure the device and lets us
> interoperate with other OSes that might want to implement this protocol.

Is this string intended to be meaningful to a human?  Or could it be 
nothing more than an ASCII version of the device structure's address?

> I've split 'cmd_submit' into four commands (control, data, isoc, irq).
> That gives us the ability to make 'data' very small.

A number of important commands are missing.  Clear-Halt springs to
mind, as do Reset-Device, Set-Configuration, and Set-Interface.

> Instead of transmitting the device number in every command, we now bind
> each socket to a particular device.  This was already what happened,
> so it was just overhead.

This isn't mentioned in your document.  There are "claim device" and 
"release device" calls, but it doesn't say anywhere that only one 
device can be claimed at a time.  Nor is there a status code for "No 
device claimed".

> There's no need to respond with the call number to each call -- the
> caller should be using the call identifier to find out what type of
> call it was.  Often the implementation will issue a command and then
> wait for the response, so even that is unnecessary.

What happens when your protocol is used with a non-reliable transport?  
Who is responsible for retransmissions/acknowledgements?

> Replying with the status is vital.

The reply to a submit call will be sent when the USB transfer is
complete.  Suppose the transfer takes a long time.  How does the client
tell the difference between a long-running transfer and an unreceived
submit?  Do you essentially assume that all protocol transfers are
reliable?

There isn't any field in the submit reply to report the status of the 
transfer (as opposed to the status of the submission).  How do errors 
like -EILSEQ get reported back to the client?

> I decided to make the 'call' value 32-bit (instead of 8-bit) to make
> everything align nicely.  Then I wanted to slim down the data command
> call, so I tucked the endpoint and direction in there too.

What about the extra flags that go with URB submissions?  
URB_NO_INTERRUPT and URB_SHORT_NOT_OK might well be useful, 
URB_ZERO_PACKET is certain to be needed, and even URB_ISO_ASAP might 
come in handy.

> I have no experience with isosynchronous transactions, nor interrupt
> transactions, so I decline to define them at this moment.

It will not be possible to support all the features of Linux's USB 
stack for these types of transfers (I'm referring to the way the 
bandwidth isn't released if a new URB is submitted during the 
completion callback).

Both types will require an additional "interval" field for submission,
and Isochronous will require "start_frame", "number_of_packets", and a
list of packet descriptors as well.  It also will require a status and
length for each component packet in the reply.

In the unlink call, what is the "seqnum" field?  Is it supposed to be 
the call identifier of the corresponding submit call?

Alan Stern

Re: USBIP protocol

[Greg KH]

On Tue, Sep 02, 2008 at 10:25:43PM -0600, Matthew Wilcox wrote:
> On Fri, Aug 29, 2008 at 07:54:07AM -0700, Greg KH wrote:
> > Fair enough, patches welcome :)
> 
> Here's a document for discussion.  No code yet, though I'm quite willing
> to modify the current usbip code to follow this new protocol.  Just a
> matter of time.
> 
> http://www.kernel.org/pub/linux/kernel/people/willy/usbip/usbip-protocol-draft-1

You should post this in the message itself, I think it is shorter than
your other comments :)

> To save some time for reviewers, here's a list of decisions I took while
> working on the document.  I appreciate that some of the decisions I made
> were not necessarily those another designer might have made, so I ask
> that any comments along the lines of "I would have done it differently"
> include a really good reason.

I agree with Alan's very good comments, and have one more:

> I have no experience with isosynchronous transactions, nor interrupt
> transactions, so I decline to define them at this moment.

Both of those are necessary, so you are going to have to define them
(mice/keyboards use interrupt and audio uses iso, so it is very common,
and easy to test for.)

> I've given up on the big/little endian thing.  Network protocols are
> traditionally BE, USB is LE and it can encapsulate SCSI which is BE again.

What do you mean, you are just going to stick with BE?

thanks,

greg k-h

Re: USBIP protocol

[Matthew Wilcox]

On Wed, Sep 03, 2008 at 11:40:21AM -0400, Alan Stern wrote:
> Most of your decisions look good to me.

Thanks for the review.

> > 'get version' is now a call rather than including a version field in
> > every request.
> 
> Have you given any thought to forward compatibility?  I imagine it's 
> too early to worry about that yet.

By forward compatibility, you mean when we rev the protocol to version 2
and try to communicate?  My thought was:

 - v1 host, v1 device -- status quo
 - v1 host, v2 device -- host will simply disconnect
 - v2 host, v1 device -- host can choose to speak v1 protocol or
   disconnect
 - v2 host, v2 device -- v2 protocol will be spoken

If we have the host transmit the highest version protocol it supports,
the device could reply with the highest version it supports.  Or we
could have the host transmit all the protocol versions it's willing to
support (eg 1, 3, 4 because we found a horrible security problem in
version 2).  It can all get quite involved.

> > Devices are now referred to as an ascii string rather than an encoded
> > 4-byte quantity.  This helps userspace configure the device and lets us
> > interoperate with other OSes that might want to implement this protocol.
> 
> Is this string intended to be meaningful to a human?  Or could it be 
> nothing more than an ASCII version of the device structure's address?

It needs to be typable by a human.  It's up to the implementation
whether it wants to report "1-1" so that "usbip --attach localhost 1-1"
continues to work.  I don't mind if I have to type "usbip --attach
localhost Jhd%s" as long as that's one of the options I get from usbip
--list.  It's not intended to be "usbip --attach 'AuthenTec, Inc. AES2501
Fingerprint Sensor'" (though I suppose a sufficiently imaginative
implementation could take any arbitrary string and try to DWIM ... that
wouldn't be prohibited).  Refusing to accept a string that came
from the list would be pretty mean though.

> > I've split 'cmd_submit' into four commands (control, data, isoc, irq).
> > That gives us the ability to make 'data' very small.
> 
> A number of important commands are missing.  Clear-Halt springs to
> mind, as do Reset-Device, Set-Configuration, and Set-Interface.

Set Configuration and Set Interface are requests on the command pipe
according to the usb 2.0 spec (these would be a control message
targetted at endpoint 0).  I don't find Reset Device in usb 2.0; in
usb-storage there's Bulk-Only Mass Storage Reset which is again a
request on the command pipe.  I also don't find Clear Halt in usb 2.0;
but if I'm reading 9.4.5 correctly, the Halt state is cleared by sending
a GetStatus control message to the appropriate endpoint.

> > Instead of transmitting the device number in every command, we now bind
> > each socket to a particular device.  This was already what happened,
> > so it was just overhead.
> 
> This isn't mentioned in your document.  There are "claim device" and 
> "release device" calls, but it doesn't say anywhere that only one 
> device can be claimed at a time.  Nor is there a status code for "No 
> device claimed".

This is true.  I'll make that more clear.  In fact, drawing a state
transition diagram might be in order.

> > There's no need to respond with the call number to each call -- the
> > caller should be using the call identifier to find out what type of
> > call it was.  Often the implementation will issue a command and then
> > wait for the response, so even that is unnecessary.
> 
> What happens when your protocol is used with a non-reliable transport?  
> Who is responsible for retransmissions/acknowledgements?

Good question, and one I hadn't been thinking about too hard since the
current implementation is over TCP.  See next answer ...

> > Replying with the status is vital.
> 
> The reply to a submit call will be sent when the USB transfer is
> complete.  Suppose the transfer takes a long time.  How does the client
> tell the difference between a long-running transfer and an unreceived
> submit?  Do you essentially assume that all protocol transfers are
> reliable?

How about this:

 - The sender has a timeout after which time, it will re-send the call.
 - If the recipient receives a call that is still outstanding, it
   replies with a status akin to NFS's EJUKEBOX that means "It's in
   progress".
 - If the recipient receives a call for a reply it has already sent, it
   just resends it.

I'd welcome some network expertise on this protocol.  Arnaldo?

> There isn't any field in the submit reply to report the status of the 
> transfer (as opposed to the status of the submission).  How do errors 
> like -EILSEQ get reported back to the client?

Isn't that error entirely within the remote device and should be retried
there?

> > I decided to make the 'call' value 32-bit (instead of 8-bit) to make
> > everything align nicely.  Then I wanted to slim down the data command
> > call, so I tucked the endpoint and direction in there too.
> 
> What about the extra flags that go with URB submissions?  
> URB_NO_INTERRUPT and URB_SHORT_NOT_OK might well be useful, 
> URB_ZERO_PACKET is certain to be needed, and even URB_ISO_ASAP might 
> come in handy.

I think including those flags in the 'call' variable would make sense,
assuming they make sense at this level.  I'm trying to understand that,
both in the context of pass-through and in the context of a pure
userspace USB device.

> > I have no experience with isosynchronous transactions, nor interrupt
> > transactions, so I decline to define them at this moment.
> 
> It will not be possible to support all the features of Linux's USB 
> stack for these types of transfers (I'm referring to the way the 
> bandwidth isn't released if a new URB is submitted during the 
> completion callback).

Right.

> Both types will require an additional "interval" field for submission,
> and Isochronous will require "start_frame", "number_of_packets", and a
> list of packet descriptors as well.  It also will require a status and
> length for each component packet in the reply.

That was my vague understanding, but I'd not looked into it really,
other than to note all those fields were unused for control and data
commands.

> In the unlink call, what is the "seqnum" field?  Is it supposed to be 
> the call identifier of the corresponding submit call?

Sorry, I thought I'd renamed it to 'victim'.  It is indeed the identifier
of the call that is to be unlinked.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Matthew Wilcox]

On Wed, Sep 03, 2008 at 08:57:06AM -0700, Greg KH wrote:
> > I have no experience with isosynchronous transactions, nor interrupt
> > transactions, so I decline to define them at this moment.
> 
> Both of those are necessary, so you are going to have to define them
> (mice/keyboards use interrupt and audio uses iso, so it is very common,
> and easy to test for.)

I don't think I have any usb audio devices.  Do you have any spares I
could borrow from you at Kernel Summit?

> > I've given up on the big/little endian thing.  Network protocols are
> > traditionally BE, USB is LE and it can encapsulate SCSI which is BE again.
> 
> What do you mean, you are just going to stick with BE?

I'm just going to use BE in the protocol, even though it's encapsulating
a LE protocol.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Alan Stern]

On Wed, 3 Sep 2008, Matthew Wilcox wrote:

> > A number of important commands are missing.  Clear-Halt springs to
> > mind, as do Reset-Device, Set-Configuration, and Set-Interface.
> 
> Set Configuration and Set Interface are requests on the command pipe
> according to the usb 2.0 spec (these would be a control message
> targetted at endpoint 0).  I don't find Reset Device in usb 2.0; in

It's referred to as SetPortFeature(PORT_RESET) in section 11.24.2.13;  
the request is actually sent to the device's upstream hub.

> usb-storage there's Bulk-Only Mass Storage Reset which is again a
> request on the command pipe.  I also don't find Clear Halt in usb 2.0;
> but if I'm reading 9.4.5 correctly, the Halt state is cleared by sending
> a GetStatus control message to the appropriate endpoint.

In fact it's cleared by sending ClearFeature(ENDPOINT_HALT) to the 
appropriate endpoint.

Never mind the details; the important point is this: All these 
requests, in addition to sending data over the USB bus, have to modify 
the host's internal state.

	Set-Configuration updates the list of available interfaces
	and endpoints.

	Set-Interface updates the list of available endpoints.

	Reset-Device requires the host to verify that the device is
	still connected to the port and its descriptors haven't
	changed.  The host also has to restore the former configuration
	and altsettings.

	Clear-Halt requires the endpoint's data toggle value in the 
	host controller to be reset.

So it isn't good enough for the client simply to send a few packets; 
the server needs to know when these events happen.  In theory they can 
be detected by parsing packets as they are sent, but IMO it would be 
better for them to be higher-level calls.


> > The reply to a submit call will be sent when the USB transfer is
> > complete.  Suppose the transfer takes a long time.  How does the client
> > tell the difference between a long-running transfer and an unreceived
> > submit?  Do you essentially assume that all protocol transfers are
> > reliable?
> 
> How about this:
> 
>  - The sender has a timeout after which time, it will re-send the call.

How will this timeout be determined?

>  - If the recipient receives a call that is still outstanding, it
>    replies with a status akin to NFS's EJUKEBOX that means "It's in
>    progress".

Should the server send a "submission accepted" message some time before
the first timeout occurs?  For long-running requests this would
eliminate a resend.

>  - If the recipient receives a call for a reply it has already sent, it
>    just resends it.

This requires the server to keep each completed reply for some time.  
When can this data be freed?


> > There isn't any field in the submit reply to report the status of the 
> > transfer (as opposed to the status of the submission).  How do errors 
> > like -EILSEQ get reported back to the client?
> 
> Isn't that error entirely within the remote device and should be retried
> there?

In brief, no.  The caller needs to know; retrying isn't always the 
appropriate response.


> > What about the extra flags that go with URB submissions?  
> > URB_NO_INTERRUPT and URB_SHORT_NOT_OK might well be useful, 
> > URB_ZERO_PACKET is certain to be needed, and even URB_ISO_ASAP might 
> > come in handy.
> 
> I think including those flags in the 'call' variable would make sense,
> assuming they make sense at this level.  I'm trying to understand that,
> both in the context of pass-through and in the context of a pure
> userspace USB device.

They do make sense at this level, just as they make sense in usbfs.
(However the point of USB_SHORT_NOT_OK would be lost; the semantics of 
that flag would need to be extended for it to be useful.)

Alan Stern

Re: USBIP protocol

[Greg KH]

On Wed, Sep 03, 2008 at 01:43:49PM -0600, Matthew Wilcox wrote:
> On Wed, Sep 03, 2008 at 08:57:06AM -0700, Greg KH wrote:
> > > I have no experience with isosynchronous transactions, nor interrupt
> > > transactions, so I decline to define them at this moment.
> > 
> > Both of those are necessary, so you are going to have to define them
> > (mice/keyboards use interrupt and audio uses iso, so it is very common,
> > and easy to test for.)
> 
> I don't think I have any usb audio devices.  Do you have any spares I
> could borrow from you at Kernel Summit?

I only have one left, and I need it for testing, sorry.  There are quite
a few cheap usb audio headsets out there if you just want to buy your
own.

> > > I've given up on the big/little endian thing.  Network protocols are
> > > traditionally BE, USB is LE and it can encapsulate SCSI which is BE again.
> > 
> > What do you mean, you are just going to stick with BE?
> 
> I'm just going to use BE in the protocol, even though it's encapsulating
> a LE protocol.

That's fine with me.

thanks,

greg k-h

Re: USBIP protocol

[Matthew Wilcox]

On Wed, Sep 03, 2008 at 04:15:39PM -0400, Alan Stern wrote:
> On Wed, 3 Sep 2008, Matthew Wilcox wrote:
> 
> > > A number of important commands are missing.  Clear-Halt springs to
> > > mind, as do Reset-Device, Set-Configuration, and Set-Interface.
> > 
> > Set Configuration and Set Interface are requests on the command pipe
> > according to the usb 2.0 spec (these would be a control message
> > targetted at endpoint 0).  I don't find Reset Device in usb 2.0; in
> 
> It's referred to as SetPortFeature(PORT_RESET) in section 11.24.2.13;  
> the request is actually sent to the device's upstream hub.

Mmm.  In this case, we don't have an upstream hub.  The vhci-hcd driver
is on the host and sends URBs across the network where they're received
by the stub driver.  The stub then sends them to the exported device.

I don't have a USB hub, so I can't tell you what happens if the exported
device is plugged into one.

What causes Linux to send a 'reset' to a hub?  What do we do in similar
circumstances if the device isn't attached to a hub?  Is there a *hci
specific way of resetting a bus that's directly attached?

(I suspect you're going to be completely right here and we need to add a
'device reset' message to the protocol.  What isn't clear to me is how
the stub on the device side is going to communicate that reset to the
device.)

> Never mind the details; the important point is this: All these 
> requests, in addition to sending data over the USB bus, have to modify 
> the host's internal state.
> 
> 	Set-Configuration updates the list of available interfaces
> 	and endpoints.
> 
> 	Set-Interface updates the list of available endpoints.
> 
> 	Reset-Device requires the host to verify that the device is
> 	still connected to the port and its descriptors haven't
> 	changed.  The host also has to restore the former configuration
> 	and altsettings.
> 
> 	Clear-Halt requires the endpoint's data toggle value in the 
> 	host controller to be reset.

I see these requests need to be understood by the vhci (host) side, and
by the device itself, but it's not clear to me that they need to be
comprehended by the stub driver on the device side.  OTOH, I'm a long
way from being an expert on the USB protocol, so please tell me again
that I'm wrong ;-)

> So it isn't good enough for the client simply to send a few packets; 
> the server needs to know when these events happen.  In theory they can 
> be detected by parsing packets as they are sent, but IMO it would be 
> better for them to be higher-level calls.

I agree with you that the client-side stub should not be parsing requests.

> > How about this:
> > 
> >  - The sender has a timeout after which time, it will re-send the call.
> 
> How will this timeout be determined?

That's an implementation not protocol question ... I suspect that
something akin to the TCP transmission rules would suffice.

> >  - If the recipient receives a call that is still outstanding, it
> >    replies with a status akin to NFS's EJUKEBOX that means "It's in
> >    progress".
> 
> Should the server send a "submission accepted" message some time before
> the first timeout occurs?  For long-running requests this would
> eliminate a resend.

I think that would be a reasonable thing to do.  Of course, the message
could be dropped on the way, so device-side implementations MAY do this,
not MUST.

> >  - If the recipient receives a call for a reply it has already sent, it
> >    just resends it.
> 
> This requires the server to keep each completed reply for some time.  
> When can this data be freed?

When the device-side receives another packet to the same endpoint,
perhaps?  Does USB permit multiple outstanding commands to the same
endpoint?

> > > There isn't any field in the submit reply to report the status of the 
> > > transfer (as opposed to the status of the submission).  How do errors 
> > > like -EILSEQ get reported back to the client?
> > 
> > Isn't that error entirely within the remote device and should be retried
> > there?
> 
> In brief, no.  The caller needs to know; retrying isn't always the 
> appropriate response.

OK, but if the transfer didn't complete, the submission doesn't have a
status either, so we can use the same status field for both protocol
errors and submission errors.

> > > What about the extra flags that go with URB submissions?  
> > > URB_NO_INTERRUPT and URB_SHORT_NOT_OK might well be useful, 
> > > URB_ZERO_PACKET is certain to be needed, and even URB_ISO_ASAP might 
> > > come in handy.
> > 
> > I think including those flags in the 'call' variable would make sense,
> > assuming they make sense at this level.  I'm trying to understand that,
> > both in the context of pass-through and in the context of a pure
> > userspace USB device.
> 
> They do make sense at this level, just as they make sense in usbfs.
> (However the point of USB_SHORT_NOT_OK would be lost; the semantics of 
> that flag would need to be extended for it to be useful.)

OK.  I think I'm going to change the 4 byte call id to be a 1 byte
call id, and 3 bytes of padding for non-submit commands.  For submit
commands, I'll use 1 byte for the direction + endpoint.  That leaves 2
bytes for flags.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Greg KH]

On Thu, Sep 04, 2008 at 03:48:02PM -0600, Matthew Wilcox wrote:
> 
> I don't have a USB hub, so I can't tell you what happens if the exported
> device is plugged into one.

I strongly suggest, if you are going to be touching USB code, that you
spend the $10 and get yourself a hub, I'm sure Intel might even allow
you to expense it.

If not, I'll take up a collection for you at the Plumbers conference :)

thanks,

greg k-h

Re: USBIP protocol

[Pete Zaitcev]

On Thu, 4 Sep 2008 15:48:02 -0600, Matthew Wilcox <matthew@wil.cx> wrote:

> > It's referred to as SetPortFeature(PORT_RESET) in section 11.24.2.13;  
> > the request is actually sent to the device's upstream hub.
> 
> Mmm.  In this case, we don't have an upstream hub.  The vhci-hcd driver
> is on the host and sends URBs across the network where they're received
> by the stub driver.  The stub then sends them to the exported device.

I'm sorry I'm barging into the conversation, but a HC without a
root hub is rather shocking. One would think it easier to emulate
a root hub with one port in HCD than to teach the whole stack about
such thing as a hubless HC.

-- Pete

Re: USBIP protocol

[Tilman Schmidt]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On Thu, 4 Sep 2008 15:48:02 -0600, Matthew Wilcox wrote:
> On Wed, Sep 03, 2008 at 04:15:39PM -0400, Alan Stern wrote:
[...]
>> It's referred to as SetPortFeature(PORT_RESET) in section 11.24.2.13;  
>> the request is actually sent to the device's upstream hub.
> 
> Mmm.  In this case, we don't have an upstream hub.  The vhci-hcd driver
> is on the host and sends URBs across the network where they're received
> by the stub driver.  The stub then sends them to the exported device.

The way I've always understood it, there is, by definition,
always an upstream hub. It's the entity controlling the port
to which the device is connected. In the case discussed here,
I guess the stub would have to play that part.

Am I wrong?

-- 
Tilman Schmidt                    E-Mail: tilman@imap.cc
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Ungeöffnet mindestens haltbar bis: (siehe Rückseite)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: USBIP protocol

[Alan Stern]

On Thu, 4 Sep 2008, Matthew Wilcox wrote:

> > It's referred to as SetPortFeature(PORT_RESET) in section 11.24.2.13;  
> > the request is actually sent to the device's upstream hub.
> 
> Mmm.  In this case, we don't have an upstream hub.  The vhci-hcd driver
> is on the host and sends URBs across the network where they're received
> by the stub driver.  The stub then sends them to the exported device.

Sometimes I find the terminology a little confusing.  "Client" and
"server" are clear enough, as is "device-side" (synonymous with
"server").  I guess vhci-hcd runs on the client, right?

But "host" is ambiguous; vhci-hcd is a host controller driver on the
client and [uoe]hci-hcd is a host controller driver on the server.  
Thus both sides are hosts.  Also, "stub" isn't too clear.  Is vhci-hcd
a stub driver?  Or does the stub driver run strictly on the server?

Getting back to your question...  All devices have an upstream hub,
except for root hubs.  In this case the device's upstream hub is the
physical hub it is plugged into if it plugged into a hub, or the
physical root hub it is plugged into otherwise.  Either way, the
upstream hub has to be managed by the server, not by the client.  
That's another reason why device reset needs to be treated as a special
call.

Similarly, suspend and resume involve sending requests to the upstream 
hub.  They too would need to be treated specially.

> I don't have a USB hub, so I can't tell you what happens if the exported
> device is plugged into one.

It shouldn't make any difference.  If it does then there's something 
wrong with the implementation.

> What causes Linux to send a 'reset' to a hub?

I'm not sure I understand the question.  Linux sends a port-reset 
request whenever anyone calls usb_reset_device().  Is that what you 
meant?

>  What do we do in similar
> circumstances if the device isn't attached to a hub?

Every device is attached to a hub, except for root hubs.  
usb_reset_device() fails if its argument is a root hub.  (Root hubs 
cannot be reset like normal devices.)

>  Is there a *hci
> specific way of resetting a bus that's directly attached?

USB doesn't reset buses; it resets devices (or more precisely, a 
device's port on its upstream hub).  So I guess you're asking if there 
is a driver-specific way of resetting a port on a root hub.  The answer 
is Yes, but it is encapsulated in a driver-independent interface.

> (I suspect you're going to be completely right here and we need to add a
> 'device reset' message to the protocol.  What isn't clear to me is how
> the stub on the device side is going to communicate that reset to the
> device.)

It will call usb_reset_device().

> > Never mind the details; the important point is this: All these 
> > requests, in addition to sending data over the USB bus, have to modify 
> > the host's internal state.
> > 
> > 	Set-Configuration updates the list of available interfaces
> > 	and endpoints.
> > 
> > 	Set-Interface updates the list of available endpoints.
> > 
> > 	Reset-Device requires the host to verify that the device is
> > 	still connected to the port and its descriptors haven't
> > 	changed.  The host also has to restore the former configuration
> > 	and altsettings.
> > 
> > 	Clear-Halt requires the endpoint's data toggle value in the 
> > 	host controller to be reset.
> 
> I see these requests need to be understood by the vhci (host) side, and
> by the device itself, but it's not clear to me that they need to be
> comprehended by the stub driver on the device side.  OTOH, I'm a long
> way from being an expert on the USB protocol, so please tell me again
> that I'm wrong ;-)

Okay.  You're wrong.  :-)

The server's USB stack needs to know about these requests because they
need to affect hardware and software state on the server.  If you tried
to pass one these things directly from the client to the device, you'd
find that suddenly things weren't working.  The hardware and software
on the server would no longer be able to communicate with the device
after the device's state was changed without the server's knowledge.


> > >  - If the recipient receives a call for a reply it has already sent, it
> > >    just resends it.
> > 
> > This requires the server to keep each completed reply for some time.  
> > When can this data be freed?
> 
> When the device-side receives another packet to the same endpoint,
> perhaps?  Does USB permit multiple outstanding commands to the same
> endpoint?

This isn't a hardware-level issue; it's a software issue.  Linux's USB 
stack allows multiple outstanding requests to be queued for the same 
endpoint.

> > > > There isn't any field in the submit reply to report the status of the 
> > > > transfer (as opposed to the status of the submission).  How do errors 
> > > > like -EILSEQ get reported back to the client?
> > > 
> > > Isn't that error entirely within the remote device and should be retried
> > > there?
> > 
> > In brief, no.  The caller needs to know; retrying isn't always the 
> > appropriate response.
> 
> OK, but if the transfer didn't complete, the submission doesn't have a
> status either, so we can use the same status field for both protocol
> errors and submission errors.

It would be easy enough to multiplex the two status values in the same
field, since URB statuses are ordinary negative Exxxx values.  If 
USBIP's errors are represented using positive integers then there would 
be no confusion.

Alan Stern

Re: USBIP protocol

[Matthew Wilcox]

[Sorry, I meant to send this mail before I left on Friday ... best laid
plans, etc]

On Fri, Sep 05, 2008 at 11:05:53AM -0400, Alan Stern wrote:
> On Thu, 4 Sep 2008, Matthew Wilcox wrote:
> 
> > > It's referred to as SetPortFeature(PORT_RESET) in section 11.24.2.13;  
> > > the request is actually sent to the device's upstream hub.
> > 
> > Mmm.  In this case, we don't have an upstream hub.  The vhci-hcd driver
> > is on the host and sends URBs across the network where they're received
> > by the stub driver.  The stub then sends them to the exported device.
> 
> Sometimes I find the terminology a little confusing.  "Client" and
> "server" are clear enough, as is "device-side" (synonymous with
> "server").  I guess vhci-hcd runs on the client, right?

'Client' and 'Server' are totally not clear.  Is the 'server' the big
machine in the closet without USB devices, or is the machine with the
device attached to it 'serving' the device to the machine without usb
devices?

(I'd like to give a big shout-out to my homies working on X Windows for
making this confusion possible)

Controller machine:

Loads vhci-hcd
runs usbip --attach machine-b 2-2

Target machine:

Loads usbip
runs usbipd
runs bind_driver --usbip 2-2

The controller machine now gets a device 9-2 which accesses the target
machine's 2-2 device.

I suppose it's my fault for not documenting this clearly in the protocol
document.  Fixed now.

> But "host" is ambiguous; vhci-hcd is a host controller driver on the
> client and [uoe]hci-hcd is a host controller driver on the server.  
> Thus both sides are hosts.  Also, "stub" isn't too clear.  Is vhci-hcd
> a stub driver?  Or does the stub driver run strictly on the server?

The usbip module is also referred to as the stub driver:

[  402.731706] usbip:Stub Driver for USB/IP:1.0

> Getting back to your question...  All devices have an upstream hub,
> except for root hubs.  In this case the device's upstream hub is the
> physical hub it is plugged into if it plugged into a hub, or the
> physical root hub it is plugged into otherwise.  Either way, the
> upstream hub has to be managed by the server, not by the client.  
> That's another reason why device reset needs to be treated as a special
> call.

OK, I was confused, I didn't realise the *hci included a hub.  I always
thought of a hub as an external device.

So the hub in the usbip scenario is on the controller machine.  Obviously,
the target machine also has a hub (ohci/uhci/ehci), but from the point
of view of sending a reset, it would be sent to vhci which then has to
communicate it to the device somehow.  So yep, let's include a reset
message in the protocol and that's how it will work.

Brian, how does it work today?  Does it just ignore reset commands?

> Similarly, suspend and resume involve sending requests to the upstream 
> hub.  They too would need to be treated specially.

An excellent point.  More commands!

> I'm not sure I understand the question.  Linux sends a port-reset 
> request whenever anyone calls usb_reset_device().  Is that what you 
> meant?

Not quite, but it's good enough for me.

> > > Never mind the details; the important point is this: All these 
> > > requests, in addition to sending data over the USB bus, have to modify 
> > > the host's internal state.
> > > 
> > > 	Set-Configuration updates the list of available interfaces
> > > 	and endpoints.
> > > 
> > > 	Set-Interface updates the list of available endpoints.
> > > 
> > > 	Reset-Device requires the host to verify that the device is
> > > 	still connected to the port and its descriptors haven't
> > > 	changed.  The host also has to restore the former configuration
> > > 	and altsettings.
> > > 
> > > 	Clear-Halt requires the endpoint's data toggle value in the 
> > > 	host controller to be reset.
> > 
> > I see these requests need to be understood by the vhci (host) side, and
> > by the device itself, but it's not clear to me that they need to be
> > comprehended by the stub driver on the device side.  OTOH, I'm a long
> > way from being an expert on the USB protocol, so please tell me again
> > that I'm wrong ;-)
> 
> Okay.  You're wrong.  :-)

You say that so authoritatively, I'm convinced!

> The server's USB stack needs to know about these requests because they
> need to affect hardware and software state on the server.  If you tried
> to pass one these things directly from the client to the device, you'd
> find that suddenly things weren't working.  The hardware and software
> on the server would no longer be able to communicate with the device
> after the device's state was changed without the server's knowledge.

I'm going to assume you mean 'device server' in this context (now
referred to as target machine).

I just don't understand why the target machine's stack needs to know
which endpoints are available.  Is it for when the controller detaches?
I would think that a USB reset should be issued to the device when
control changes between a real driver (eg usb-storage) and the stub
driver.

> > > >  - If the recipient receives a call for a reply it has already sent, it
> > > >    just resends it.
> > > 
> > > This requires the server to keep each completed reply for some time.  
> > > When can this data be freed?
> > 
> > When the device-side receives another packet to the same endpoint,
> > perhaps?  Does USB permit multiple outstanding commands to the same
> > endpoint?
> 
> This isn't a hardware-level issue; it's a software issue.  Linux's USB 
> stack allows multiple outstanding requests to be queued for the same 
> endpoint.

I guess the question is where the queueing happens -- does it happen in
the *hci driver, or does it happen in the device?  Either way, we could
choose whether to queue the urbs on the controller machine or on the
target machine -- it's a complexity / performance tradeoff.  I don't
know where the correct balance point is.

> > OK, but if the transfer didn't complete, the submission doesn't have a
> > status either, so we can use the same status field for both protocol
> > errors and submission errors.
> 
> It would be easy enough to multiplex the two status values in the same
> field, since URB statuses are ordinary negative Exxxx values.  If 
> USBIP's errors are represented using positive integers then there would 
> be no confusion.

Works for me.  We'll still want to document what they are since we
presumably want to encourage people to come up with their own
implementations of this protocol for other operating systems and fake
devices and so on.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: USBIP protocol

[Steve Calfee]

Hi all,

I have been looking at the USBIP stuff and don't understand the
generic probe issues.

In your second patch:
static struct usb_device_id stub_table[] = {
+#if 0
+	/* just an example */
+	{ USB_DEVICE(0x05ac, 0x0301) },   /* Mac 1 button mouse */
+	{ USB_DEVICE(0x0430, 0x0009) },   /* Plat Home Keyboard */
+	{ USB_DEVICE(0x059b, 0x0001) },   /* Iomega USB Zip 100 */
+	{ USB_DEVICE(0x04b3, 0x4427) },   /* IBM USB CD-ROM */
+	{ USB_DEVICE(0x05a9, 0xa511) },   /* LifeView USB cam */
+	{ USB_DEVICE(0x55aa, 0x0201) },   /* Imation card reader */
+	{ USB_DEVICE(0x046d, 0x0870) },   /* Qcam Express(QV-30) */
+	{ USB_DEVICE(0x04bb, 0x0101) },   /* IO-DATA HD 120GB */
+	{ USB_DEVICE(0x04bb, 0x0904) },   /* IO-DATA USB-ET/TX */
+	{ USB_DEVICE(0x04bb, 0x0201) },   /* IO-DATA USB-ET/TX */
+	{ USB_DEVICE(0x08bb, 0x2702) },   /* ONKYO USB Speaker */
+	{ USB_DEVICE(0x046d, 0x08b2) },   /* Logicool Qcam 4000 Pro */
+#endif
+	/* magic for wild card */
+	{ .driver_info = 1 },
+	{ 0, }                                     /* Terminating entry */
+};

You have changed your driver to a wild card for everything. Fine, it
makes sense, your probe routine should check if it wants a new device
and fail if it does not want it.

My problem is I am using the latest Suse enterprise linux which uses
kernel 2.6.18..... In that kernel the probes are done by the device
system and if a modprobe definition says the module is for this device
it gets loaded (by udev I think) and probed. fine. If the module is
already loaded the kernel probes the modules in the order they were
insmoded (if they say they want the particular VID/PID/CLASS/wildcard
etc.)

The problem is the probing is done by the kernel driver stuff and the
insmoding and user space intelligence is done by udev. Once a kernel
module is installed it must take all devices that match it's device ID
table.  Remember USBIP matches everything! That is because (at least
in the 2.6.18 kernel) the kernel driver stuff stops probing after a
probe fails. This is completely counter-intuitive to me, I would have
assumed that if a probe routine fails, the driver system would keep
going through the list looking for a driver that wants it.

Is this rule still true in the current kernels (I will care if and
when Novel/Suse updates its next enterprise addition).

So, at least in the latest Suse enterprise linux, these are the rules:
1) Any current driver will be probed in the order of insmod. If it
fails, the device will not have a driver
2) If there is no driver, udev will modprobe one and that process will
attach it to the device because all un-probed devices will be
reprobed.

I have not found a way to reprobe unclaimed drivers (and don't know
what order a reprobe would use - if USBIP rejects a device again
because it (USBIP_ is insmoded first it would still be unclaimed).

So to be more descriptive, say you want to do a remote USBIP flash
drive for drive "B", out of three drives, "A", "B","C". How can I
(from user space) enable this sequence? I can control the order of
module loading, (sort of).  I need usb_storage active twice and USBIP
active once for "B". This cannot not work with 2.6.18 - does it work
now?

Regards, Steve

Re: USBIP protocol

[Greg KH]

On Tue, Sep 09, 2008 at 12:12:37AM -0700, Steve Calfee wrote:
> 
> My problem is I am using the latest Suse enterprise linux which uses
> kernel 2.6.18

Actually it's 2.6.16 "based" with over 2000 patches added to it :)

> ..... In that kernel the probes are done by the device
> system and if a modprobe definition says the module is for this device
> it gets loaded (by udev I think) and probed. fine. If the module is
> already loaded the kernel probes the modules in the order they were
> insmoded (if they say they want the particular VID/PID/CLASS/wildcard
> etc.)
> 
> The problem is the probing is done by the kernel driver stuff and the
> insmoding and user space intelligence is done by udev. Once a kernel
> module is installed it must take all devices that match it's device ID
> table.  Remember USBIP matches everything! That is because (at least
> in the 2.6.18 kernel) the kernel driver stuff stops probing after a
> probe fails.

No it doesn't, it continues on.  Or it should.  Perhaps it's been fixed
since then, I would rely on that kernel to be looking like anything
"real" these days, it is very old and only supported by Novell, and not
the target audience for this patch at all.

thanks,

greg k-h

Re: USBIP protocol

[Greg KH]

On Tue, Sep 09, 2008 at 12:33:56AM -0700, Greg KH wrote:
> On Tue, Sep 09, 2008 at 12:12:37AM -0700, Steve Calfee wrote:
> > 
> > My problem is I am using the latest Suse enterprise linux which uses
> > kernel 2.6.18
> 
> Actually it's 2.6.16 "based" with over 2000 patches added to it :)
> 
> > ..... In that kernel the probes are done by the device
> > system and if a modprobe definition says the module is for this device
> > it gets loaded (by udev I think) and probed. fine. If the module is
> > already loaded the kernel probes the modules in the order they were
> > insmoded (if they say they want the particular VID/PID/CLASS/wildcard
> > etc.)
> > 
> > The problem is the probing is done by the kernel driver stuff and the
> > insmoding and user space intelligence is done by udev. Once a kernel
> > module is installed it must take all devices that match it's device ID
> > table.  Remember USBIP matches everything! That is because (at least
> > in the 2.6.18 kernel) the kernel driver stuff stops probing after a
> > probe fails.
> 
> No it doesn't, it continues on.  Or it should.  Perhaps it's been fixed
> since then, I would rely on that kernel to be looking like anything
                          ^ not

> "real" these days, it is very old and only supported by Novell, and not
> the target audience for this patch at all.

Re: USBIP protocol

[Alan Stern]

On Mon, 8 Sep 2008, Matthew Wilcox wrote:

> [Sorry, I meant to send this mail before I left on Friday ... best laid
> plans, etc]

That's okay; I'm not on a deadline.  :-)

> > Sometimes I find the terminology a little confusing.  "Client" and
> > "server" are clear enough, as is "device-side" (synonymous with
> > "server").  I guess vhci-hcd runs on the client, right?
> 
> 'Client' and 'Server' are totally not clear.  Is the 'server' the big
> machine in the closet without USB devices, or is the machine with the
> device attached to it 'serving' the device to the machine without usb
> devices?

It is unfortunate that these two words have been used in multiple ways, 
with meanings that are sometimes only subtly different and sometimes 
grossly conflicting.

Isn't it standard in networking circles for "server" to mean the system
running a daemon that waits for incoming connections and "client" to
mean the system running an on-demand program that initiates a
connection?

> (I'd like to give a big shout-out to my homies working on X Windows for
> making this confusion possible)
> 
> Controller machine:
> 
> Loads vhci-hcd
> runs usbip --attach machine-b 2-2
> 
> Target machine:
> 
> Loads usbip
> runs usbipd
> runs bind_driver --usbip 2-2
> 
> The controller machine now gets a device 9-2 which accesses the target
> machine's 2-2 device.
> 
> I suppose it's my fault for not documenting this clearly in the protocol
> document.  Fixed now.

Okay, "controller" and "target".  I'm not sure that "controller" is the
best word either, since it might be confused with "USB host
controller".

> > But "host" is ambiguous; vhci-hcd is a host controller driver on the
> > client and [uoe]hci-hcd is a host controller driver on the server.  
> > Thus both sides are hosts.  Also, "stub" isn't too clear.  Is vhci-hcd
> > a stub driver?  Or does the stub driver run strictly on the server?
> 
> The usbip module is also referred to as the stub driver:
> 
> [  402.731706] usbip:Stub Driver for USB/IP:1.0

Hmmm.  Above you said that the controller system runs a program named 
"usbip" and the target system loads a module named "usbip".  Since the 
stub driver is the usbip _module_, it must run on the target (AKA 
server).  Right?

> > Getting back to your question...  All devices have an upstream hub,
> > except for root hubs.  In this case the device's upstream hub is the
> > physical hub it is plugged into if it plugged into a hub, or the
> > physical root hub it is plugged into otherwise.  Either way, the
> > upstream hub has to be managed by the server, not by the client.  
> > That's another reason why device reset needs to be treated as a special
> > call.
> 
> OK, I was confused, I didn't realise the *hci included a hub.  I always
> thought of a hub as an external device.

Yes; it is referred to as a "virtual root hub", or just "root hub" for 
short.  We didn't make this up; section 10.2.8 of the USB spec is 
entitled "Root Hub".

FYI, the *hci-hcd modules are collectively known as "host controller
drivers", or HCDs for short.  We USB guys sling that term around a lot
-- if you start using it too you'll impress other programmers.  :-)


> > The server's USB stack needs to know about these requests because they
> > need to affect hardware and software state on the server.  If you tried
> > to pass one these things directly from the client to the device, you'd
> > find that suddenly things weren't working.  The hardware and software
> > on the server would no longer be able to communicate with the device
> > after the device's state was changed without the server's knowledge.
> 
> I'm going to assume you mean 'device server' in this context (now
> referred to as target machine).

Right.

> I just don't understand why the target machine's stack needs to know
> which endpoints are available.  Is it for when the controller detaches?

"Detaches" isn't a USB term.  Do you mean it in the sense of a SCSI 
initiator detaching from its target?

In any event, the answer is that each endpoint has several properties
associated with it (direction, type, maximum packet size, and so on),
and these properties have to be known by both the target's USB stack
and the target's USB host controller hardware.  The hardware uses DMA
to access various in-memory data structures describing the I/O
operations it should perform, and the format and contents of these data
structures depends on the endpoints' properties.  (To make matters
worse, the hardware will sometimes maintain an internal cache for parts
of the data structures and some of the properties; when the properties 
change the cache must be flushed.)

With new configurations or altsettings you end up with a new set of
endpoints, or the same old endpoints with a new set of properties.  
Thus the target really needs to know when these things change.

> I would think that a USB reset should be issued to the device when
> control changes between a real driver (eg usb-storage) and the stub
> driver.

I suppose the controller system, as part of its normal initialization 
procedure for new devices, would issue such a reset.  This area is 
still a little cloudy -- we know that the controller system shouldn't 
treat a new USBIP device exactly the same as a new normal USB device, 
but exactly what the difference should be has yet to be fleshed out.

We may even want to have some sort of method table with entries for 
initialization, reset, suspend, resume, and clear-halt.

> > > When the device-side receives another packet to the same endpoint,
> > > perhaps?  Does USB permit multiple outstanding commands to the same
> > > endpoint?
> > 
> > This isn't a hardware-level issue; it's a software issue.  Linux's USB 
> > stack allows multiple outstanding requests to be queued for the same 
> > endpoint.
> 
> I guess the question is where the queueing happens -- does it happen in
> the *hci driver, or does it happen in the device?

It happens in the HCD, not in the device.

>  Either way, we could
> choose whether to queue the urbs on the controller machine or on the
> target machine -- it's a complexity / performance tradeoff.  I don't
> know where the correct balance point is.

There's more to it.  The USB stack behaves qualitatively and 
quantitatively differently in these two cases:

	An URB completes, the completion handler returns, and then
	a new URB is submitted.

	A new URB is submitted either before the old one completes
	or while the old URB's completion handler is running.

With USBIP it's not feasible for an URB queued on the controller
machine to be submitted as part of the target's completion handler
processing.  If the URB was queued in the target's usbipd driver then 
this would be feasible.  And of course, if the URB was queued in the 
target's HCD then no special treatment would be needed.

For relevant background information, read the kerneldoc for 
usb_submit_urb() and usb_unlink_urb() in drivers/usb/core/urb.c.

Alan Stern