Re: [PATCH V13 3/5] libxl: add pvusb API

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 1/27/2016 at 12:21 AM, in message
<CAFLBxZaTWuViBUjH663tR-GWfwRGCWSNKoRecik9+Q6TJ3HQmA@mail.gmail.com>, George
Dunlap <George.Dunlap@eu.citrix.com> wrote: 
> On Tue, Jan 26, 2016 at 7:43 AM, Chun Yan Liu <cyliu@suse.com> wrote: 
> > 
> > 
> >>>> On 1/20/2016 at 12:56 PM, in message 
> > <569F83F5020000660009E6AC@relay2.provo.novell.com>, "Chun Yan Liu" 
> > <cyliu@suse.com> wrote: 
> > 
> >> 
> >>>>> On 1/19/2016 at 11:48 PM, in message 
> >> <22174.23240.402164.635740@mariner.uk.xensource.com>, Ian Jackson 
> >> <Ian.Jackson@eu.citrix.com> wrote: 
> >> > Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"): 
> >> > > Add pvusb APIs, including: 
> >> > >  - attach/detach (create/destroy) virtual usb controller. 
> >> > >  - attach/detach usb device 
> >> > >  - list usb controller and usb devices 
> >> > >  - some other helper functions 
> >> > 
> >> > 
> >> > Thanks.  This is making progress but I'm afraid we're not quite there 
> >> > yet. 
> >> > 
> >> > 
> >> > > +static int usbback_dev_unassign(libxl__gc *gc, const char *busid) 
> >> > > +{ 
> >> > ... 
> >> > > +    /* Till here, USB device has been unbound from USBBACK and 
> >> > > +     * removed from xenstore, usb list couldn't show it anymore, 
> >> > > +     * so no matter removimg driver path successfully or not, 
> >> > > +     * we will report operation success. 
> >> > > +     */ 
> >> > 
> >> > I'm still unconvinced by this and this may mean that the code in this 
> >> > function is in the wrong order.  Earlier we had this exchange: 
> >> > 
> >> > > > Ought this function to really report success if these calls fail ? 
> >> > > 
> >> > > I think so. Till here, the USB device has already been unbound from 
> >> > > usbback and removed from xenstore. usb-list cannot list it any more. 
> >> > 
> >> > The problem is that I think that if this function fails, it can leave 
> >> >  - debris in xenstore (the usbback path) 
> >> Yes, it's true. 
> >> 
> >> >  - the interface bound to the wrong driver 
> >> No, it won't be bound to 'wrong' driver, only maybe not bound to any driver 
> >> (Already unbound from usbback, but failed to rebound to its original 
> >> driver). 
> >> In this case, we would report warning: failed to rebind to driver xxx. 
> >> 
> >> > And then there is no way for the user to get libxl to re-attempt the 
> >> > operation, or clean up.  Am I right ? 
> >> 
> >> Yes. No way to re-attempt usbdev-detach or cleanup driver path in 
> >> xenstore. But won't affect next time usbdev-attach the same device. 
> >> 
> >> > 
> >> > One way to avoid this kind of problem is to deal with the xenstore 
> >> > path last.  That way the device will still appear as attached to the 
> >> > domain. 
> >> 
> >> I'm afraid if the side effect is acceptable? In my testing, some USB
> >> bluetooth 
> >> device always fails to rebind to 'btusb' driver after it's unbound 
> >> from 'usbback'. In this case, we can't detach it from the domain then. 

George & Ian, may I have your thoughts? Except for above case, I think dealing
with xenstore at last place is indeed better.
PS: I'll take vocation for half month, so hope to make any progress before that :-)

Thanks,
Chunyan

> > 
> > Ian J., any opinion on this? If it's still thought to be better, I'll  
> update patch. 
>  
> I think Ian may be waiting for me to reply and express an opinion; but 
> unfortunately that will have to wait until next week. :-( 
>  
>  -George 
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb API"):
> There's a difference between "making it intelligent" and "not making it
> broken". :-)  Given that users can potentially cause a number of these
> things to fail just by pressing Ctrl-C, we need to at least make sure
> that we don't get into a completely wedged state that the user can't do
> anything to fix.  That requires some careful thought.

Right.  There is a useful design principle which can help simplify and
clarify: "crash-only software".[1][2]

The idea is that when an error occurs, it is usually best to simply
stop and leave the system in whatever intermediate state.  The next
run of the software is responsible for discovering, interpreting and
if necessary rectifying the situation.

In general these recovery paths are necessary anyway.  So the
on-error-cleanup doesn't help.


In the context of that series, I think that means:

We observe that there is an ordering of possible states
      attached to dom0 driver
      unattached
      attached to usbback
      assigned to a guest

When reconfiguring a device, it will move through these states in
order (forwards or backwards, normally).

NB that I don't discuss here what information may be recorded in
xenstore and the VM configuration at each stage: so in fact there are
various micro-states in between the major states shown above (eg,
"attached to usbback and in process of attaching to geust").


To make a coherent design, we need to:

Arrange that each of these states can be seen by the user somehow.
(Eg in output from list-assignable.)

Arrange that each intermediate state can be recovered to at least one
endpoint state by use of some appropriate command(s).

Arrange that the recording of metadata in xenstore and the domain json
config occurs in the right order to support the above (ie, that the
micro-states are right).  After an approach has been chosen, to show
that the design is correct, it is probably easiest to explicitly
enumerate all the micro-states.


Ideally I would like to see this aspect of the design covered in a doc
comment (or maybe commit messages), in some form or other.


I hope this is of some help.

Thanks,
Ian.


[1] https://www.usenix.org/events/hotos03/tech/full_papers/candea/candea.pdf
[2] https://en.wikipedia.org/wiki/Crash-only_software

Re: [PATCH V13 3/5] libxl: add pvusb API

[George Dunlap]

On 04/02/16 14:39, Juergen Gross wrote:
> On 04/02/16 02:53, Chun Yan Liu wrote:
>>
>>
>>>>> On 2/3/2016 at 10:33 PM, in message <56B20FCC.3010308@citrix.com>, George
>> Dunlap <george.dunlap@citrix.com> wrote: 
>>> On 02/02/16 18:11, Ian Jackson wrote: 
>>>> George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb  
>>> API"): 
>>>>> There are effectively four states a device can be in, from the 
>>>>> 'assignment' point of view: 
>>>>>
>>>>> 1. Assigned to the normal Linux device driver(s) for the USB device 
>>>>>
>>>>> 2. Assigned to no driver 
>>>>>
>>>>> 3. Assigned to usbback, but not yet assigned to any guest 
>>>>>
>>>>> 4. Assigned to a guest 
>>>>  
>>>> Thanks for your clear explanation (of which I will snip much). 
>>>>  
>>>>> Additionally, each USB "device" has one or more "interfaces".  To 
>>>>> assign a "device" to usbback in the sysfs case means assigning all of 
>>>>> the "interfaces".  The code seems to assume that different interfaces 
>>>>> from the same device can be assigned to different drivers. 
>>>>  
>>>> It is indeed the case that in principle a single USB device with 
>>>> multiple interfaces can be assigned to multiple different drivers. 
>>>>  
>>>>> Regarding Ian's comments: 
>>>>>
>>>>> Since "assigned to the guest" and "listed under the pvusb xenstore 
>>>>> node" are the same thing, is it even *possible* to (safely) unassign 
>>>>> the device from usbback before removing the xenstore nodes? 
>>>>  
>>>> It might be possible to remove some of the xenstore nodes but leave 
>>>> others present, so that usbback detaches, but enough information 
>>>> remains for libxl to know that Xen still `owns' the device. 
>>>>  
>>>> But, surely usbback needs to cope with the notion that the device 
>>>> might disappear.  USB devices can disappear at any time. 
>>>  
>>> That's true. But if the USB device has actually disappeared, the device 
>>> is in fact "safe" from usbback.  I think I might consider state 2 safe 
>>> to go to, but I definitely wouldn't consider state 1 safe with the 
>>> xenstore nodes still in place. 
>>>  
>>>>  
>>>>> It's not clear to me under what conditions 3->2 might fail, 
>>>>  
>>>> As an example, someone might press ^C on `xl usb-detach'. 
>>>  
>>> *grumble* 
>>>  
>>>>> or what could be done about it if it did fail. 
>>>>  
>>>> The most obvious reason for it failing is that something somewhere 
>>>> still held onto the device.  (For umounting filesystems, and detaching 
>>>> block devices, this happens a lot.)  So if the detach from usbback 
>>>> fails would probably be possible to simply retry it. 
>>>  
>>> I'm not sure what this means in the context of moving from 3 (assigned 
>>> to usbback) to 2 (unassigned).  usbback will definitely not use the 
>>> device to mount something in dom0; and I'm pretty sure has no visibility 
>>> as to whether it's being used as a filesystem in the domU. 
>>>  
>>> (Moving from 1 -> 2 of course would be subject to this sort of thing, 
>>> but that's a different issue.) 
>>>  
>>>>> Regarding 2->1, again it's not clear that there's anything libxl can 
>>>>> do.  Reloading the driver's module might reset it enough to pick up 
>>>>> the (now unassigned) USB devices again; other than that, rebooting is 
>>>>> probably the best option. 
>>>>  
>>>> I think re-attempting the bind may work.  USB devices can be flaky. 
>>>>  
>>>>> It's also not clear to me, if some functions succeed in being 
>>>>> reassigned and others fail, whether it's best to just try to assign as 
>>>>> many as we can, or better to go back and un-assign all the ones that 
>>>>> succeeded. 
>>>>  
>>>> Unless explicitly requested, I don't think the system should create 
>>>> situations some interfaces are assigned to host drivers and some to 
>>>> usbback. 
>>>  
>>> I was talking here about whether it would be better to have some 
>>> interfaces assigned to the original drivers and some interfaces left 
>>> unassigned, or to try to leave all interfaces unassigned if any of the 
>>> assignments fail. 
>>>  
>>> I agree, it would be better to try to avoid the possibility of having 
>>> some interfaces bound to usbback and some interfaces bound to the 
>>> original drivers. 
>>>  
>>>> And, I'm a fan of `crash-only software': in general, if a failure 
>>>> occurs, the situation should just be left as-is.  The intermediate 
>>>> state needs to be visible and rectifiable. 
>>>>  
>>>> This approach to software state machines avoids bugs where the system 
>>>> gets into `impossible' states, recovery from which is impossible 
>>>> because the designers didn't anticipate them. 
>>>>  
>>>> It would be tolerable if the recovery sometimes involves `lsusb' and 
>>>> echoing things into sysfs, but it would be better if not. 
>>>  
>>> Right -- so what about this.  When removing a USB device: 
>>>  
>>> * First modify the pvusb xenstore nodes such that 1) it's safe to 
>>> attempt removing the interfaces from usbback, but 2) they still show up 
>>> in usb-list.  (This may be a noop.) 
>>>  
>>> * Attempt to remove all interfaces from usbback; if any of these fail, 
>>> stop and report an error.  Possible recovery options: 
>>>  - Re-try the usb_remove command 
>>>  - Re-load usbback (obviously disruptive to other VMs) 
>>>  - Reboot the host 
>>>  - Manually try unbinding with sysfs 
>>>  
>>> * Remove the remaining pvusb xenstore nodes.  If this fails, stop and 
>>> report the error.  Possible recovery options: 
>>>  - Re-try the usb_remove command 
>>>  
>>> = REBIND OPTION 1: Do the best we can without extra commands 
>>>  
>>> * Attempt to re-bind the interfaces to the original drivers, as recorded 
>>> in the libxl usb xenstore nodes.  If any of these fail, report an error 
>>> but continue to try the rest of the interfaces, and remove the xenstore 
>>> nodes containing information about the original drivers.  Possible 
>>> recovery options for the user: 
>>>  - Reload the original drivers 
>>>  - Reboot the host 
>>>  - Manually try rebinding with sysfs 
>>>  
>>> = REBIND OPTION 2: Include a recovery command, usb-retry-rebind 
>>>  
>>> * Attempt to re-bind the interfaces to the original drivers, as recorded 
>>> in the libxl usb xenstore nodes.  If any of these fail, stop immediately 
>>> and report an error; do not remove the xenstore nodes containing the 
>>> original drivers of any interfaces that failed to rebind.  Possible 
>>> recovery options for the user: 
>>>  - Run 'xl usbdev-retry-rebind', which will just execute this step again 
>>>  - Unload and reload original host drivers 
>>>  - Reboot the host 
>>>  
>>> Both of these: 
>>>  
>>> 1. Will avoid the state of some interfaces bound to usbback, some 
>>> rebound to the original drivers 
>>>  
>>> 2. Give the user a convenient way to re-try unbinding from usbback it failed 
>>>  
>>> 3. Give the user out-of-xl ways to attempt to recover the state other 
>>> than messing around with sysfs. 
>>>  
>>> Rebind option 2 will give the user a convenient way to retry rebinding 
>>> to the original driver via xl if that step failed. 
>>>  
>>> I'm inclined to suggest rebind option #1 for now, just to keep things 
>>> simple. 
>>>  
>>> Thoughts? 
>>>  
>>> Chunyan, would the first half of that (removing from usbback before 
>>> removing the pvusb xenstore nodes) actually work? 
>>
>> From testing, yes, it works. But I am not sure if it is safe though. Juergen
>> should be very clear about that according to usbback internal codes.
>>
>> Juergen, could you help to confirm that?
> 
> I think it should work.
> 
> TBH, I don't think the error handling of driver binding/unbinding should
> be made too intelligent. In the long run I hope to have a qemu-based
> pvUSB backend. And in this case driver binding/unbinding will be handled
> completely by qemu/libusb.
> 
> Whether we want to keep the kernel based pvUSB-backend support in xl
> at the end should be discussed later. I'll either add the qemu based
> variant or I'll replace the kernel variant with the qemu one as soon
> as the qemu backend has been accepted.

There's a difference between "making it intelligent" and "not making it
broken". :-)  Given that users can potentially cause a number of these
things to fail just by pressing Ctrl-C, we need to at least make sure
that we don't get into a completely wedged state that the user can't do
anything to fix.  That requires some careful thought.

 -George

Re: [PATCH V13 3/5] libxl: add pvusb API

[Juergen Gross]

On 04/02/16 02:53, Chun Yan Liu wrote:
> 
> 
>>>> On 2/3/2016 at 10:33 PM, in message <56B20FCC.3010308@citrix.com>, George
> Dunlap <george.dunlap@citrix.com> wrote: 
>> On 02/02/16 18:11, Ian Jackson wrote: 
>>> George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb  
>> API"): 
>>>> There are effectively four states a device can be in, from the 
>>>> 'assignment' point of view: 
>>>>
>>>> 1. Assigned to the normal Linux device driver(s) for the USB device 
>>>>
>>>> 2. Assigned to no driver 
>>>>
>>>> 3. Assigned to usbback, but not yet assigned to any guest 
>>>>
>>>> 4. Assigned to a guest 
>>>  
>>> Thanks for your clear explanation (of which I will snip much). 
>>>  
>>>> Additionally, each USB "device" has one or more "interfaces".  To 
>>>> assign a "device" to usbback in the sysfs case means assigning all of 
>>>> the "interfaces".  The code seems to assume that different interfaces 
>>>> from the same device can be assigned to different drivers. 
>>>  
>>> It is indeed the case that in principle a single USB device with 
>>> multiple interfaces can be assigned to multiple different drivers. 
>>>  
>>>> Regarding Ian's comments: 
>>>>
>>>> Since "assigned to the guest" and "listed under the pvusb xenstore 
>>>> node" are the same thing, is it even *possible* to (safely) unassign 
>>>> the device from usbback before removing the xenstore nodes? 
>>>  
>>> It might be possible to remove some of the xenstore nodes but leave 
>>> others present, so that usbback detaches, but enough information 
>>> remains for libxl to know that Xen still `owns' the device. 
>>>  
>>> But, surely usbback needs to cope with the notion that the device 
>>> might disappear.  USB devices can disappear at any time. 
>>  
>> That's true. But if the USB device has actually disappeared, the device 
>> is in fact "safe" from usbback.  I think I might consider state 2 safe 
>> to go to, but I definitely wouldn't consider state 1 safe with the 
>> xenstore nodes still in place. 
>>  
>>>  
>>>> It's not clear to me under what conditions 3->2 might fail, 
>>>  
>>> As an example, someone might press ^C on `xl usb-detach'. 
>>  
>> *grumble* 
>>  
>>>> or what could be done about it if it did fail. 
>>>  
>>> The most obvious reason for it failing is that something somewhere 
>>> still held onto the device.  (For umounting filesystems, and detaching 
>>> block devices, this happens a lot.)  So if the detach from usbback 
>>> fails would probably be possible to simply retry it. 
>>  
>> I'm not sure what this means in the context of moving from 3 (assigned 
>> to usbback) to 2 (unassigned).  usbback will definitely not use the 
>> device to mount something in dom0; and I'm pretty sure has no visibility 
>> as to whether it's being used as a filesystem in the domU. 
>>  
>> (Moving from 1 -> 2 of course would be subject to this sort of thing, 
>> but that's a different issue.) 
>>  
>>>> Regarding 2->1, again it's not clear that there's anything libxl can 
>>>> do.  Reloading the driver's module might reset it enough to pick up 
>>>> the (now unassigned) USB devices again; other than that, rebooting is 
>>>> probably the best option. 
>>>  
>>> I think re-attempting the bind may work.  USB devices can be flaky. 
>>>  
>>>> It's also not clear to me, if some functions succeed in being 
>>>> reassigned and others fail, whether it's best to just try to assign as 
>>>> many as we can, or better to go back and un-assign all the ones that 
>>>> succeeded. 
>>>  
>>> Unless explicitly requested, I don't think the system should create 
>>> situations some interfaces are assigned to host drivers and some to 
>>> usbback. 
>>  
>> I was talking here about whether it would be better to have some 
>> interfaces assigned to the original drivers and some interfaces left 
>> unassigned, or to try to leave all interfaces unassigned if any of the 
>> assignments fail. 
>>  
>> I agree, it would be better to try to avoid the possibility of having 
>> some interfaces bound to usbback and some interfaces bound to the 
>> original drivers. 
>>  
>>> And, I'm a fan of `crash-only software': in general, if a failure 
>>> occurs, the situation should just be left as-is.  The intermediate 
>>> state needs to be visible and rectifiable. 
>>>  
>>> This approach to software state machines avoids bugs where the system 
>>> gets into `impossible' states, recovery from which is impossible 
>>> because the designers didn't anticipate them. 
>>>  
>>> It would be tolerable if the recovery sometimes involves `lsusb' and 
>>> echoing things into sysfs, but it would be better if not. 
>>  
>> Right -- so what about this.  When removing a USB device: 
>>  
>> * First modify the pvusb xenstore nodes such that 1) it's safe to 
>> attempt removing the interfaces from usbback, but 2) they still show up 
>> in usb-list.  (This may be a noop.) 
>>  
>> * Attempt to remove all interfaces from usbback; if any of these fail, 
>> stop and report an error.  Possible recovery options: 
>>  - Re-try the usb_remove command 
>>  - Re-load usbback (obviously disruptive to other VMs) 
>>  - Reboot the host 
>>  - Manually try unbinding with sysfs 
>>  
>> * Remove the remaining pvusb xenstore nodes.  If this fails, stop and 
>> report the error.  Possible recovery options: 
>>  - Re-try the usb_remove command 
>>  
>> = REBIND OPTION 1: Do the best we can without extra commands 
>>  
>> * Attempt to re-bind the interfaces to the original drivers, as recorded 
>> in the libxl usb xenstore nodes.  If any of these fail, report an error 
>> but continue to try the rest of the interfaces, and remove the xenstore 
>> nodes containing information about the original drivers.  Possible 
>> recovery options for the user: 
>>  - Reload the original drivers 
>>  - Reboot the host 
>>  - Manually try rebinding with sysfs 
>>  
>> = REBIND OPTION 2: Include a recovery command, usb-retry-rebind 
>>  
>> * Attempt to re-bind the interfaces to the original drivers, as recorded 
>> in the libxl usb xenstore nodes.  If any of these fail, stop immediately 
>> and report an error; do not remove the xenstore nodes containing the 
>> original drivers of any interfaces that failed to rebind.  Possible 
>> recovery options for the user: 
>>  - Run 'xl usbdev-retry-rebind', which will just execute this step again 
>>  - Unload and reload original host drivers 
>>  - Reboot the host 
>>  
>> Both of these: 
>>  
>> 1. Will avoid the state of some interfaces bound to usbback, some 
>> rebound to the original drivers 
>>  
>> 2. Give the user a convenient way to re-try unbinding from usbback it failed 
>>  
>> 3. Give the user out-of-xl ways to attempt to recover the state other 
>> than messing around with sysfs. 
>>  
>> Rebind option 2 will give the user a convenient way to retry rebinding 
>> to the original driver via xl if that step failed. 
>>  
>> I'm inclined to suggest rebind option #1 for now, just to keep things 
>> simple. 
>>  
>> Thoughts? 
>>  
>> Chunyan, would the first half of that (removing from usbback before 
>> removing the pvusb xenstore nodes) actually work? 
> 
> From testing, yes, it works. But I am not sure if it is safe though. Juergen
> should be very clear about that according to usbback internal codes.
> 
> Juergen, could you help to confirm that?

I think it should work.

TBH, I don't think the error handling of driver binding/unbinding should
be made too intelligent. In the long run I hope to have a qemu-based
pvUSB backend. And in this case driver binding/unbinding will be handled
completely by qemu/libusb.

Whether we want to keep the kernel based pvUSB-backend support in xl
at the end should be discussed later. I'll either add the qemu based
variant or I'll replace the kernel variant with the qemu one as soon
as the qemu backend has been accepted.


Juergen

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 2/3/2016 at 10:33 PM, in message <56B20FCC.3010308@citrix.com>, George
Dunlap <george.dunlap@citrix.com> wrote: 
> On 02/02/16 18:11, Ian Jackson wrote: 
> > George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb  
> API"): 
> >> There are effectively four states a device can be in, from the 
> >> 'assignment' point of view: 
> >> 
> >> 1. Assigned to the normal Linux device driver(s) for the USB device 
> >> 
> >> 2. Assigned to no driver 
> >> 
> >> 3. Assigned to usbback, but not yet assigned to any guest 
> >> 
> >> 4. Assigned to a guest 
> >  
> > Thanks for your clear explanation (of which I will snip much). 
> >  
> >> Additionally, each USB "device" has one or more "interfaces".  To 
> >> assign a "device" to usbback in the sysfs case means assigning all of 
> >> the "interfaces".  The code seems to assume that different interfaces 
> >> from the same device can be assigned to different drivers. 
> >  
> > It is indeed the case that in principle a single USB device with 
> > multiple interfaces can be assigned to multiple different drivers. 
> >  
> >> Regarding Ian's comments: 
> >> 
> >> Since "assigned to the guest" and "listed under the pvusb xenstore 
> >> node" are the same thing, is it even *possible* to (safely) unassign 
> >> the device from usbback before removing the xenstore nodes? 
> >  
> > It might be possible to remove some of the xenstore nodes but leave 
> > others present, so that usbback detaches, but enough information 
> > remains for libxl to know that Xen still `owns' the device. 
> >  
> > But, surely usbback needs to cope with the notion that the device 
> > might disappear.  USB devices can disappear at any time. 
>  
> That's true. But if the USB device has actually disappeared, the device 
> is in fact "safe" from usbback.  I think I might consider state 2 safe 
> to go to, but I definitely wouldn't consider state 1 safe with the 
> xenstore nodes still in place. 
>  
> >  
> >> It's not clear to me under what conditions 3->2 might fail, 
> >  
> > As an example, someone might press ^C on `xl usb-detach'. 
>  
> *grumble* 
>  
> >> or what could be done about it if it did fail. 
> >  
> > The most obvious reason for it failing is that something somewhere 
> > still held onto the device.  (For umounting filesystems, and detaching 
> > block devices, this happens a lot.)  So if the detach from usbback 
> > fails would probably be possible to simply retry it. 
>  
> I'm not sure what this means in the context of moving from 3 (assigned 
> to usbback) to 2 (unassigned).  usbback will definitely not use the 
> device to mount something in dom0; and I'm pretty sure has no visibility 
> as to whether it's being used as a filesystem in the domU. 
>  
> (Moving from 1 -> 2 of course would be subject to this sort of thing, 
> but that's a different issue.) 
>  
> >> Regarding 2->1, again it's not clear that there's anything libxl can 
> >> do.  Reloading the driver's module might reset it enough to pick up 
> >> the (now unassigned) USB devices again; other than that, rebooting is 
> >> probably the best option. 
> >  
> > I think re-attempting the bind may work.  USB devices can be flaky. 
> >  
> >> It's also not clear to me, if some functions succeed in being 
> >> reassigned and others fail, whether it's best to just try to assign as 
> >> many as we can, or better to go back and un-assign all the ones that 
> >> succeeded. 
> >  
> > Unless explicitly requested, I don't think the system should create 
> > situations some interfaces are assigned to host drivers and some to 
> > usbback. 
>  
> I was talking here about whether it would be better to have some 
> interfaces assigned to the original drivers and some interfaces left 
> unassigned, or to try to leave all interfaces unassigned if any of the 
> assignments fail. 
>  
> I agree, it would be better to try to avoid the possibility of having 
> some interfaces bound to usbback and some interfaces bound to the 
> original drivers. 
>  
> > And, I'm a fan of `crash-only software': in general, if a failure 
> > occurs, the situation should just be left as-is.  The intermediate 
> > state needs to be visible and rectifiable. 
> >  
> > This approach to software state machines avoids bugs where the system 
> > gets into `impossible' states, recovery from which is impossible 
> > because the designers didn't anticipate them. 
> >  
> > It would be tolerable if the recovery sometimes involves `lsusb' and 
> > echoing things into sysfs, but it would be better if not. 
>  
> Right -- so what about this.  When removing a USB device: 
>  
> * First modify the pvusb xenstore nodes such that 1) it's safe to 
> attempt removing the interfaces from usbback, but 2) they still show up 
> in usb-list.  (This may be a noop.) 
>  
> * Attempt to remove all interfaces from usbback; if any of these fail, 
> stop and report an error.  Possible recovery options: 
>  - Re-try the usb_remove command 
>  - Re-load usbback (obviously disruptive to other VMs) 
>  - Reboot the host 
>  - Manually try unbinding with sysfs 
>  
> * Remove the remaining pvusb xenstore nodes.  If this fails, stop and 
> report the error.  Possible recovery options: 
>  - Re-try the usb_remove command 
>  
> = REBIND OPTION 1: Do the best we can without extra commands 
>  
> * Attempt to re-bind the interfaces to the original drivers, as recorded 
> in the libxl usb xenstore nodes.  If any of these fail, report an error 
> but continue to try the rest of the interfaces, and remove the xenstore 
> nodes containing information about the original drivers.  Possible 
> recovery options for the user: 
>  - Reload the original drivers 
>  - Reboot the host 
>  - Manually try rebinding with sysfs 
>  
> = REBIND OPTION 2: Include a recovery command, usb-retry-rebind 
>  
> * Attempt to re-bind the interfaces to the original drivers, as recorded 
> in the libxl usb xenstore nodes.  If any of these fail, stop immediately 
> and report an error; do not remove the xenstore nodes containing the 
> original drivers of any interfaces that failed to rebind.  Possible 
> recovery options for the user: 
>  - Run 'xl usbdev-retry-rebind', which will just execute this step again 
>  - Unload and reload original host drivers 
>  - Reboot the host 
>  
> Both of these: 
>  
> 1. Will avoid the state of some interfaces bound to usbback, some 
> rebound to the original drivers 
>  
> 2. Give the user a convenient way to re-try unbinding from usbback it failed 
>  
> 3. Give the user out-of-xl ways to attempt to recover the state other 
> than messing around with sysfs. 
>  
> Rebind option 2 will give the user a convenient way to retry rebinding 
> to the original driver via xl if that step failed. 
>  
> I'm inclined to suggest rebind option #1 for now, just to keep things 
> simple. 
>  
> Thoughts? 
>  
> Chunyan, would the first half of that (removing from usbback before 
> removing the pvusb xenstore nodes) actually work? 

>From testing, yes, it works. But I am not sure if it is safe though. Juergen
should be very clear about that according to usbback internal codes.

Juergen, could you help to confirm that?

Thanks,
Chunyan

>  
>  -George 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 2/3/2016 at 10:38 PM, in message <56B210FA.7040803@citrix.com>, George
Dunlap <george.dunlap@citrix.com> wrote: 
> On 03/02/16 07:34, Chun Yan Liu wrote: 
> >  
> >  
> >>>> On 2/3/2016 at 02:11 AM, in message 
> > <22192.61775.427189.268007@mariner.uk.xensource.com>, Ian Jackson 
> > <Ian.Jackson@eu.citrix.com> wrote:  
> >> George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb  
> API"):  
> >>> There are effectively four states a device can be in, from the  
> >>> 'assignment' point of view:  
> >>>   
> >>> 1. Assigned to the normal Linux device driver(s) for the USB device  
> >>>   
> >>> 2. Assigned to no driver  
> >>>   
> >>> 3. Assigned to usbback, but not yet assigned to any guest  
> >>>   
> >>> 4. Assigned to a guest  
> >>   
> >> Thanks for your clear explanation (of which I will snip much).  
> >>   
> >>> Additionally, each USB "device" has one or more "interfaces".  To  
> >>> assign a "device" to usbback in the sysfs case means assigning all of  
> >>> the "interfaces".  The code seems to assume that different interfaces  
> >>> from the same device can be assigned to different drivers.  
> >>   
> >> It is indeed the case that in principle a single USB device with  
> >> multiple interfaces can be assigned to multiple different drivers.  
> >>   
> >>> Regarding Ian's comments:  
> >>>   
> >>> Since "assigned to the guest" and "listed under the pvusb xenstore  
> >>> node" are the same thing, is it even *possible* to (safely) unassign  
> >>> the device from usbback before removing the xenstore nodes?  
> >>   
> >> It might be possible to remove some of the xenstore nodes but leave  
> >> others present, so that usbback detaches, but enough information  
> >> remains for libxl to know that Xen still `owns' the device.  
> >  
> > Indeed "unassign from usbback, but listed under pvusb xenstore" is 
> > a confusing state. usb-list can list it but guest can not see it.  
> > What user can do under that state is: reattempt usbdev_remove, if it  
> > succeeds, everything is cleaned up, that's the best result; but  
> > possibly it still fails (for example, in my testing, always cannot  
> > rebind to original driver), in this case, the confusing state will  
> > be lasting, and the device could not be removed, this is worse. 
>  
> As I said in my other mail, I think removing the pvusb nodes should be 
> done once it's successfully un-bound from usbback, *even if* the re-bind 
> to the original driver fails.  (That is, once it reaches state 2, 
> usb-list should no longer list it.) 
>  
> >>> Perhaps the best approach code-wise is to change the "goto out" on  
> >>> failure of unbind_usbintf() into a "continue".  That way:  
> >>>   
> >>> 1. All interfaces which can be re-assigned are re-assigned (and work  
> >>> as much as possible)  
> >>>   
> >>> 2. All interfaces which can be unbound but not re-assigned are at  
> >>> least unbound (so that reloading the original driver might pick them  
> >>> up)  
> >>   
> >> I certainly don't mind the software trying to do as much of its task  
> >> as possible. 
> >  
> > Could I understand that this way is acceptable? That means: removing  
> > xenstore, and as much as we could (on failure of "unbind from usbback" 
> > or "bind to original driver", don't "goto out", just "continue"). 
>  
> I think part of it depends on what is *possible*.  If it's possible to 
> safely unbind the device from usbback while retaining its place in the 
> pvusb-related xenstore nodes, then I think we should (so that the user

Juergen, I think you are the person who can answer the question best.
Can you confirm that?

-Chunyan

> can re-try removing it).  If it's not possible, then of course we have 
> to remove the pvusb xenstore nodes first, and then we'll just have to 
> deal as gracefully as possible with failure unbinding from usbback. 
>  
>  -George 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[George Dunlap]

On 03/02/16 07:34, Chun Yan Liu wrote:
> 
> 
>>>> On 2/3/2016 at 02:11 AM, in message
> <22192.61775.427189.268007@mariner.uk.xensource.com>, Ian Jackson
> <Ian.Jackson@eu.citrix.com> wrote: 
>> George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb API"): 
>>> There are effectively four states a device can be in, from the 
>>> 'assignment' point of view: 
>>>  
>>> 1. Assigned to the normal Linux device driver(s) for the USB device 
>>>  
>>> 2. Assigned to no driver 
>>>  
>>> 3. Assigned to usbback, but not yet assigned to any guest 
>>>  
>>> 4. Assigned to a guest 
>>  
>> Thanks for your clear explanation (of which I will snip much). 
>>  
>>> Additionally, each USB "device" has one or more "interfaces".  To 
>>> assign a "device" to usbback in the sysfs case means assigning all of 
>>> the "interfaces".  The code seems to assume that different interfaces 
>>> from the same device can be assigned to different drivers. 
>>  
>> It is indeed the case that in principle a single USB device with 
>> multiple interfaces can be assigned to multiple different drivers. 
>>  
>>> Regarding Ian's comments: 
>>>  
>>> Since "assigned to the guest" and "listed under the pvusb xenstore 
>>> node" are the same thing, is it even *possible* to (safely) unassign 
>>> the device from usbback before removing the xenstore nodes? 
>>  
>> It might be possible to remove some of the xenstore nodes but leave 
>> others present, so that usbback detaches, but enough information 
>> remains for libxl to know that Xen still `owns' the device. 
> 
> Indeed "unassign from usbback, but listed under pvusb xenstore" is
> a confusing state. usb-list can list it but guest can not see it. 
> What user can do under that state is: reattempt usbdev_remove, if it 
> succeeds, everything is cleaned up, that's the best result; but 
> possibly it still fails (for example, in my testing, always cannot 
> rebind to original driver), in this case, the confusing state will 
> be lasting, and the device could not be removed, this is worse.

As I said in my other mail, I think removing the pvusb nodes should be
done once it's successfully un-bound from usbback, *even if* the re-bind
to the original driver fails.  (That is, once it reaches state 2,
usb-list should no longer list it.)

>>> Perhaps the best approach code-wise is to change the "goto out" on 
>>> failure of unbind_usbintf() into a "continue".  That way: 
>>>  
>>> 1. All interfaces which can be re-assigned are re-assigned (and work 
>>> as much as possible) 
>>>  
>>> 2. All interfaces which can be unbound but not re-assigned are at 
>>> least unbound (so that reloading the original driver might pick them 
>>> up) 
>>  
>> I certainly don't mind the software trying to do as much of its task 
>> as possible.
> 
> Could I understand that this way is acceptable? That means: removing 
> xenstore, and as much as we could (on failure of "unbind from usbback"
> or "bind to original driver", don't "goto out", just "continue").

I think part of it depends on what is *possible*.  If it's possible to
safely unbind the device from usbback while retaining its place in the
pvusb-related xenstore nodes, then I think we should (so that the user
can re-try removing it).  If it's not possible, then of course we have
to remove the pvusb xenstore nodes first, and then we'll just have to
deal as gracefully as possible with failure unbinding from usbback.

 -George

Re: [PATCH V13 3/5] libxl: add pvusb API

[George Dunlap]

On 02/02/16 18:11, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb API"):
>> There are effectively four states a device can be in, from the
>> 'assignment' point of view:
>>
>> 1. Assigned to the normal Linux device driver(s) for the USB device
>>
>> 2. Assigned to no driver
>>
>> 3. Assigned to usbback, but not yet assigned to any guest
>>
>> 4. Assigned to a guest
> 
> Thanks for your clear explanation (of which I will snip much).
> 
>> Additionally, each USB "device" has one or more "interfaces".  To
>> assign a "device" to usbback in the sysfs case means assigning all of
>> the "interfaces".  The code seems to assume that different interfaces
>> from the same device can be assigned to different drivers.
> 
> It is indeed the case that in principle a single USB device with
> multiple interfaces can be assigned to multiple different drivers.
> 
>> Regarding Ian's comments:
>>
>> Since "assigned to the guest" and "listed under the pvusb xenstore
>> node" are the same thing, is it even *possible* to (safely) unassign
>> the device from usbback before removing the xenstore nodes?
> 
> It might be possible to remove some of the xenstore nodes but leave
> others present, so that usbback detaches, but enough information
> remains for libxl to know that Xen still `owns' the device.
> 
> But, surely usbback needs to cope with the notion that the device
> might disappear.  USB devices can disappear at any time.

That's true. But if the USB device has actually disappeared, the device
is in fact "safe" from usbback.  I think I might consider state 2 safe
to go to, but I definitely wouldn't consider state 1 safe with the
xenstore nodes still in place.

> 
>> It's not clear to me under what conditions 3->2 might fail,
> 
> As an example, someone might press ^C on `xl usb-detach'.

*grumble*

>> or what could be done about it if it did fail.
> 
> The most obvious reason for it failing is that something somewhere
> still held onto the device.  (For umounting filesystems, and detaching
> block devices, this happens a lot.)  So if the detach from usbback
> fails would probably be possible to simply retry it.

I'm not sure what this means in the context of moving from 3 (assigned
to usbback) to 2 (unassigned).  usbback will definitely not use the
device to mount something in dom0; and I'm pretty sure has no visibility
as to whether it's being used as a filesystem in the domU.

(Moving from 1 -> 2 of course would be subject to this sort of thing,
but that's a different issue.)

>> Regarding 2->1, again it's not clear that there's anything libxl can
>> do.  Reloading the driver's module might reset it enough to pick up
>> the (now unassigned) USB devices again; other than that, rebooting is
>> probably the best option.
> 
> I think re-attempting the bind may work.  USB devices can be flaky.
> 
>> It's also not clear to me, if some functions succeed in being
>> reassigned and others fail, whether it's best to just try to assign as
>> many as we can, or better to go back and un-assign all the ones that
>> succeeded.
> 
> Unless explicitly requested, I don't think the system should create
> situations some interfaces are assigned to host drivers and some to
> usbback.

I was talking here about whether it would be better to have some
interfaces assigned to the original drivers and some interfaces left
unassigned, or to try to leave all interfaces unassigned if any of the
assignments fail.

I agree, it would be better to try to avoid the possibility of having
some interfaces bound to usbback and some interfaces bound to the
original drivers.

> And, I'm a fan of `crash-only software': in general, if a failure
> occurs, the situation should just be left as-is.  The intermediate
> state needs to be visible and rectifiable.
> 
> This approach to software state machines avoids bugs where the system
> gets into `impossible' states, recovery from which is impossible
> because the designers didn't anticipate them.
> 
> It would be tolerable if the recovery sometimes involves `lsusb' and
> echoing things into sysfs, but it would be better if not.

Right -- so what about this.  When removing a USB device:

* First modify the pvusb xenstore nodes such that 1) it's safe to
attempt removing the interfaces from usbback, but 2) they still show up
in usb-list.  (This may be a noop.)

* Attempt to remove all interfaces from usbback; if any of these fail,
stop and report an error.  Possible recovery options:
 - Re-try the usb_remove command
 - Re-load usbback (obviously disruptive to other VMs)
 - Reboot the host
 - Manually try unbinding with sysfs

* Remove the remaining pvusb xenstore nodes.  If this fails, stop and
report the error.  Possible recovery options:
 - Re-try the usb_remove command

= REBIND OPTION 1: Do the best we can without extra commands

* Attempt to re-bind the interfaces to the original drivers, as recorded
in the libxl usb xenstore nodes.  If any of these fail, report an error
but continue to try the rest of the interfaces, and remove the xenstore
nodes containing information about the original drivers.  Possible
recovery options for the user:
 - Reload the original drivers
 - Reboot the host
 - Manually try rebinding with sysfs

= REBIND OPTION 2: Include a recovery command, usb-retry-rebind

* Attempt to re-bind the interfaces to the original drivers, as recorded
in the libxl usb xenstore nodes.  If any of these fail, stop immediately
and report an error; do not remove the xenstore nodes containing the
original drivers of any interfaces that failed to rebind.  Possible
recovery options for the user:
 - Run 'xl usbdev-retry-rebind', which will just execute this step again
 - Unload and reload original host drivers
 - Reboot the host

Both of these:

1. Will avoid the state of some interfaces bound to usbback, some
rebound to the original drivers

2. Give the user a convenient way to re-try unbinding from usbback it failed

3. Give the user out-of-xl ways to attempt to recover the state other
than messing around with sysfs.

Rebind option 2 will give the user a convenient way to retry rebinding
to the original driver via xl if that step failed.

I'm inclined to suggest rebind option #1 for now, just to keep things
simple.

Thoughts?

Chunyan, would the first half of that (removing from usbback before
removing the pvusb xenstore nodes) actually work?

 -George

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 2/3/2016 at 12:48 AM, in message
<CAFLBxZY38HfouhD4eS63+HieQ8dfsza=rY7r_CNGhknmVLowJg@mail.gmail.com>, George
Dunlap <George.Dunlap@eu.citrix.com> wrote: 
> On Tue, Jan 19, 2016 at 3:48 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote: 
> > Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"): 
> >> > Ought this function to really report success if these calls fail ? 
> >> 
> >> I think so. Till here, the USB device has already been unbound from 
> >> usbback and removed from xenstore. usb-list cannot list it any more. 
>  
> Also, I think we probably should return some sort of error, so that 
> scripts &c can at least know something out of the ordinary has 
> happened, even if it doesn't go back to the state we started in when 
> the function was called.

That makes sense. But do we need to differentiate failure at removing
xenstore information step and error in bind/unbind step? Or, treat 
them as same? That affects doing it within other process (like 
usbctrl-detach). For the former, xl usb-list can list usb device and
one can do usbdev-detach again; and if in a usbctrl-detach process,
will stop the process. For the later, one cannot do usbdev-detach, 
and if in usbctrl-detach process, in current codes, will ignore that.

Chunyan

>  
>  -George 
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 2/3/2016 at 02:11 AM, in message
<22192.61775.427189.268007@mariner.uk.xensource.com>, Ian Jackson
<Ian.Jackson@eu.citrix.com> wrote: 
> George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb API"): 
> > There are effectively four states a device can be in, from the 
> > 'assignment' point of view: 
> >  
> > 1. Assigned to the normal Linux device driver(s) for the USB device 
> >  
> > 2. Assigned to no driver 
> >  
> > 3. Assigned to usbback, but not yet assigned to any guest 
> >  
> > 4. Assigned to a guest 
>  
> Thanks for your clear explanation (of which I will snip much). 
>  
> > Additionally, each USB "device" has one or more "interfaces".  To 
> > assign a "device" to usbback in the sysfs case means assigning all of 
> > the "interfaces".  The code seems to assume that different interfaces 
> > from the same device can be assigned to different drivers. 
>  
> It is indeed the case that in principle a single USB device with 
> multiple interfaces can be assigned to multiple different drivers. 
>  
> > Regarding Ian's comments: 
> >  
> > Since "assigned to the guest" and "listed under the pvusb xenstore 
> > node" are the same thing, is it even *possible* to (safely) unassign 
> > the device from usbback before removing the xenstore nodes? 
>  
> It might be possible to remove some of the xenstore nodes but leave 
> others present, so that usbback detaches, but enough information 
> remains for libxl to know that Xen still `owns' the device. 

Indeed "unassign from usbback, but listed under pvusb xenstore" is
a confusing state. usb-list can list it but guest can not see it. 
What user can do under that state is: reattempt usbdev_remove, if it 
succeeds, everything is cleaned up, that's the best result; but 
possibly it still fails (for example, in my testing, always cannot 
rebind to original driver), in this case, the confusing state will 
be lasting, and the device could not be removed, this is worse.

>  
> But, surely usbback needs to cope with the notion that the device 
> might disappear.  USB devices can disappear at any time. 
>  
> > It's not clear to me under what conditions 3->2 might fail, 
>  
> As an example, someone might press ^C on `xl usb-detach'. 
>  
> > or what could be done about it if it did fail. 
>  
> The most obvious reason for it failing is that something somewhere 
> still held onto the device.  (For umounting filesystems, and detaching 
> block devices, this happens a lot.)  So if the detach from usbback 
> fails would probably be possible to simply retry it. 
>  
> > Regarding 2->1, again it's not clear that there's anything libxl can 
> > do.  Reloading the driver's module might reset it enough to pick up 
> > the (now unassigned) USB devices again; other than that, rebooting is 
> > probably the best option. 
>  
> I think re-attempting the bind may work.  USB devices can be flaky. 
>  
> > It's also not clear to me, if some functions succeed in being 
> > reassigned and others fail, whether it's best to just try to assign as 
> > many as we can, or better to go back and un-assign all the ones that 
> > succeeded. 
>  
> Unless explicitly requested, I don't think the system should create 
> situations some interfaces are assigned to host drivers and some to 
> usbback. 
>  
> And, I'm a fan of `crash-only software': in general, if a failure 
> occurs, the situation should just be left as-is.  The intermediate 
> state needs to be visible and rectifiable. 
>  
> This approach to software state machines avoids bugs where the system 
> gets into `impossible' states, recovery from which is impossible 
> because the designers didn't anticipate them. 
>  
> It would be tolerable if the recovery sometimes involves `lsusb' and 
> echoing things into sysfs, but it would be better if not. 
>  
> > Perhaps the best approach code-wise is to change the "goto out" on 
> > failure of unbind_usbintf() into a "continue".  That way: 
> >  
> > 1. All interfaces which can be re-assigned are re-assigned (and work 
> > as much as possible) 
> >  
> > 2. All interfaces which can be unbound but not re-assigned are at 
> > least unbound (so that reloading the original driver might pick them 
> > up) 
>  
> I certainly don't mind the software trying to do as much of its task 
> as possible.

Could I understand that this way is acceptable? That means: removing 
xenstore, and as much as we could (on failure of "unbind from usbback"
or "bind to original driver", don't "goto out", just "continue").

Chunyan
>  
> Ian. 
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] [PATCH V13 3/5] libxl: add pvusb API"):
> There are effectively four states a device can be in, from the
> 'assignment' point of view:
> 
> 1. Assigned to the normal Linux device driver(s) for the USB device
> 
> 2. Assigned to no driver
> 
> 3. Assigned to usbback, but not yet assigned to any guest
> 
> 4. Assigned to a guest

Thanks for your clear explanation (of which I will snip much).

> Additionally, each USB "device" has one or more "interfaces".  To
> assign a "device" to usbback in the sysfs case means assigning all of
> the "interfaces".  The code seems to assume that different interfaces
> from the same device can be assigned to different drivers.

It is indeed the case that in principle a single USB device with
multiple interfaces can be assigned to multiple different drivers.

> Regarding Ian's comments:
> 
> Since "assigned to the guest" and "listed under the pvusb xenstore
> node" are the same thing, is it even *possible* to (safely) unassign
> the device from usbback before removing the xenstore nodes?

It might be possible to remove some of the xenstore nodes but leave
others present, so that usbback detaches, but enough information
remains for libxl to know that Xen still `owns' the device.

But, surely usbback needs to cope with the notion that the device
might disappear.  USB devices can disappear at any time.

> It's not clear to me under what conditions 3->2 might fail,

As an example, someone might press ^C on `xl usb-detach'.

> or what could be done about it if it did fail.

The most obvious reason for it failing is that something somewhere
still held onto the device.  (For umounting filesystems, and detaching
block devices, this happens a lot.)  So if the detach from usbback
fails would probably be possible to simply retry it.

> Regarding 2->1, again it's not clear that there's anything libxl can
> do.  Reloading the driver's module might reset it enough to pick up
> the (now unassigned) USB devices again; other than that, rebooting is
> probably the best option.

I think re-attempting the bind may work.  USB devices can be flaky.

> It's also not clear to me, if some functions succeed in being
> reassigned and others fail, whether it's best to just try to assign as
> many as we can, or better to go back and un-assign all the ones that
> succeeded.

Unless explicitly requested, I don't think the system should create
situations some interfaces are assigned to host drivers and some to
usbback.

And, I'm a fan of `crash-only software': in general, if a failure
occurs, the situation should just be left as-is.  The intermediate
state needs to be visible and rectifiable.

This approach to software state machines avoids bugs where the system
gets into `impossible' states, recovery from which is impossible
because the designers didn't anticipate them.

It would be tolerable if the recovery sometimes involves `lsusb' and
echoing things into sysfs, but it would be better if not.

> Perhaps the best approach code-wise is to change the "goto out" on
> failure of unbind_usbintf() into a "continue".  That way:
> 
> 1. All interfaces which can be re-assigned are re-assigned (and work
> as much as possible)
> 
> 2. All interfaces which can be unbound but not re-assigned are at
> least unbound (so that reloading the original driver might pick them
> up)

I certainly don't mind the software trying to do as much of its task
as possible.

Ian.

Re: [PATCH V13 3/5] libxl: add pvusb API

[George Dunlap]

On Tue, Jan 19, 2016 at 3:48 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"):
>> > Ought this function to really report success if these calls fail ?
>>
>> I think so. Till here, the USB device has already been unbound from
>> usbback and removed from xenstore. usb-list cannot list it any more.

Also, I think we probably should return some sort of error, so that
scripts &c can at least know something out of the ordinary has
happened, even if it doesn't go back to the state we started in when
the function was called.

 -George

Re: [PATCH V13 3/5] libxl: add pvusb API

[George Dunlap]

On Tue, Jan 19, 2016 at 3:48 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"):
>> +static int usbback_dev_unassign(libxl__gc *gc, const char *busid)
>> +{
> ...
>> +    /* Till here, USB device has been unbound from USBBACK and
>> +     * removed from xenstore, usb list couldn't show it anymore,
>> +     * so no matter removimg driver path successfully or not,
>> +     * we will report operation success.
>> +     */
>
> I'm still unconvinced by this and this may mean that the code in this
> function is in the wrong order.  Earlier we had this exchange:
>
>> > Ought this function to really report success if these calls fail ?
>>
>> I think so. Till here, the USB device has already been unbound from
>> usbback and removed from xenstore. usb-list cannot list it any more.
>
> The problem is that I think that if this function fails, it can leave
>  - debris in xenstore (the usbback path)
>  - the interface bound to the wrong driver
> And then there is no way for the user to get libxl to re-attempt the
> operation, or clean up.  Am I right ?
>
> One way to avoid this kind of problem is to deal with the xenstore
> path last.  That way the device will still appear as attached to the
> domain.
>
> To do this properly I think bind_usbintf may need to become idempotent
> even in the face of other callers racing to assign the device.  I
> think that would mean bind_usbif it would have to know what driver to
> expect to find the device bound to.
>
> George, am I right here ?

There are effectively four states a device can be in, from the
'assignment' point of view:

1. Assigned to the normal Linux device driver(s) for the USB device

2. Assigned to no driver

3. Assigned to usbback, but not yet assigned to any guest

4. Assigned to a guest

Regardless of the state, the USB device is visible to lsusb, and
always exists in the usb controller's device heirarchy, regardless of
what driver it's assigned to (if any).

As far as sysfs and lspci goes, pci is an exact analog.

For usb, libxl really only has two states: assigned (state 4) and
unassigned (states 1 or 2).  libxl__device_usbdev_add() will take
devices from states 1 or 2 and move them into state 3, then state 4.
libxl__device_usbdev_remove() will take devices from state 4, through
state 3, and then to state 2 (and then possibly state 1, if it has the
appropriate information).  usbdev_list will only list things in state
4.

Additionally, each USB "device" has one or more "interfaces".  To
assign a "device" to usbback in the sysfs case means assigning all of
the "interfaces".  The code seems to assume that different interfaces
from the same device can be assigned to different drivers.

For comparison, the libxl pci pass-through code has three states: not
assignable (states 1 or 2), assignable (state 3), and assigned (state
4).  libxl__device_pci_assignable_add moves it from 1 or 2 to state 3;
libxl_device_pci_add moves it from state 3 to 4 (unless the "seize"
flag is set, in which case it will call
libxl__device_pci_assignable_add if necessary).
libxl_device_pci_assignable_list lists things in state 3;
libxl_device_pci_list list things in state 4.

pci devices also have analogs called "functions"; but libxl treats
each of these separately (i.e., you actually assign a "function" to a
VM, not a "device").  If a device has several functions which all need
to be assigned, it's up to the user to assign each one individually.

So for removal, we have to consider what to do with a failure at each
of the steps.

* 4 -> 3.  This is just removing it from xenstore.  The xenstore
removal is transactional, so (I think) a failure here should mean that
it's still in state 4; it should still show up in usb_list, and the
user could try removing again if she wants.

* 3 -> 2. At this point, what the code actually does is try to unbind
each interface from usbback; if any of the calls to unbind_usbintf()
fail, it will give up (leaving any remaining functions bound to
usbback).

* 2 -> 1.  If we stored a driver path in xenstore (under the /libxl
node, not under the pvusb node), we then try to rebind each interface
to that driver.  If this binding fails, we simply print a warning and
continue (i.e., we attempt to re-bind with all interfaces, even if one
of them fails).

Regarding Ian's comments:

Since "assigned to the guest" and "listed under the pvusb xenstore
node" are the same thing, is it even *possible* to (safely) unassign
the device from usbback before removing the xenstore nodes?

It's not clear to me under what conditions 3->2 might fail, or what
could be done about it if it did fail.  Perhaps removing the usbback
module?  Failing that, rebooting the system is the only recovery
option I can think of.  Certainly trying to assign it back to the
guest (i.e., move it back to the previous legal state of '4') isn't a
good idea.

I do agree, however, that stopping in the middle here (leaving some
interfaces assigned to usbback and some potentially already
re-assigned to the original driver) is a bad; as is leaving the original
driver for the interface in xenstore.

Regarding 2->1, again it's not clear that there's anything libxl can
do.  Reloading the driver's module might reset it enough to pick up
the (now unassigned) USB devices again; other than that, rebooting is
probably the best option.

It's also not clear to me, if some functions succeed in being
reassigned and others fail, whether it's best to just try to assign as
many as we can, or better to go back and un-assign all the ones that
succeeded.

Perhaps the best approach code-wise is to change the "goto out" on
failure of unbind_usbintf() into a "continue".  That way:

1. All interfaces which can be re-assigned are re-assigned (and work
as much as possible)

2. All interfaces which can be unbound but not re-assigned are at
least unbound (so that reloading the original driver might pick them
up)

3. The xenstore paths storing the original driver information are
cleaned up.

And we should of course include some useful error messages to give the
user a hint as to what they can do to fix things; perhaps:

"ERROR: Unexpected failure, device now in wedged state (partially
assigned to usbback), don't know how to recover.  Try unloading and
reloading the usbback module, or rebooting the system."

"ERROR: Unexpected failure re-assigning device, don't know how to
recover.  Try unloading and reloading $MODULE, or rebooting the
system."

Thoughts?

 -George

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 1/27/2016 at 12:12 AM, in message <20160126161253.GA9905@gmail.com>, Olaf
Hering <olaf@aepfle.de> wrote: 
> On Tue, Jan 19, Chunyan Liu wrote: 
>  
> > +++ b/tools/libxl/libxl.c 
> > @@ -3204,7 +3204,7 @@ void  
> libxl__device_disk_local_initiate_detach(libxl__egc *egc, 
> >          aodev->dev = device; 
> >          aodev->callback = local_device_detach_cb; 
> >          aodev->force = 0; 
> > -        libxl__initiate_device_remove(egc, aodev); 
> > +        libxl__initiate_device_generic_remove(egc, aodev); 
> >          return; 
> >      } 
> >   
> > @@ -4172,8 +4172,10 @@ out: 
> >   * libxl_device_vkb_destroy 
> >   * libxl_device_vfb_remove 
> >   * libxl_device_vfb_destroy 
> > + * libxl_device_usbctrl_remove 
> > + * libxl_device_usbctrl_destroy 
>  
> This should be moved down to DEFINE_DEVICE_REMOVE_CUSTOM. 
>  
> >   */ 
> > -#define DEFINE_DEVICE_REMOVE(type, removedestroy, f)                    \ 
> > +#define DEFINE_DEVICE_REMOVE_EXT(type, remtype, removedestroy, f)        \ 
> >      int libxl_device_##type##_##removedestroy(libxl_ctx *ctx,           \ 
> >          uint32_t domid, libxl_device_##type *type,                      \ 
> >          const libxl_asyncop_how *ao_how)                                \ 
> > @@ -4193,13 +4195,19 @@ out: 
> >          aodev->dev = device;                                            \ 
> >          aodev->callback = device_addrm_aocomplete;                      \ 
> >          aodev->force = f;                                               \ 
> > -        libxl__initiate_device_remove(egc, aodev);                      \ 
> > +        libxl__initiate_device_##remtype##_remove(egc, aodev);          \ 
> >                                                                          \ 
> >      out:                                                                \ 
> > -        if (rc) return AO_CREATE_FAIL(rc);                                   
>   \ 
> > +        if (rc) return AO_CREATE_FAIL(rc);                              \ 
> >          return AO_INPROGRESS;                                           \ 
> >      } 
> >   
> > +#define DEFINE_DEVICE_REMOVE(type, removedestroy, f) \ 
> > +    DEFINE_DEVICE_REMOVE_EXT(type, generic, removedestroy, f) 
> > + 
> > +#define DEFINE_DEVICE_REMOVE_CUSTOM(type, removedestroy, f)  \ 
> > +    DEFINE_DEVICE_REMOVE_EXT(type, type, removedestroy, f) 
> > + 
> >  /* Define all remove/destroy functions and undef the macro */ 
> >   
> >  /* disk */ 
>  
>  
> If this is the way to move forward, please split this out into a 
> separate change which can be applied to staging independent of any pvusb 
> changes. 
>  
> I think the patch needs also the #undef. 
>  

Got you. Will update.

>  
> > @@ -4223,6 +4231,10 @@ DEFINE_DEVICE_REMOVE(vfb, destroy, 1) 
> >  DEFINE_DEVICE_REMOVE(vtpm, remove, 0) 
> >  DEFINE_DEVICE_REMOVE(vtpm, destroy, 1) 
> >   
> > +/* usbctrl */ 
> > +DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, remove, 0) 
> > +DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, destroy, 1) 
> > + 
> >  /* channel/console hotunplug is not implemented. There are 2  
> possibilities: 
> >   * 1. add support for secondary consoles to xenconsoled 
> >   * 2. dynamically add/remove qemu chardevs via qmp messages. */ 
>  
> A comment should mention both libxl_device_usbctrl_remove/destroy and 
> libxl__initiate_device_usbctrl_remove/destroy. 

Will add comments.

Thanks,
Chunyan

>  
> Olaf 
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[George Dunlap]

On Tue, Jan 26, 2016 at 7:43 AM, Chun Yan Liu <cyliu@suse.com> wrote:
>
>
>>>> On 1/20/2016 at 12:56 PM, in message
> <569F83F5020000660009E6AC@relay2.provo.novell.com>, "Chun Yan Liu"
> <cyliu@suse.com> wrote:
>
>>
>>>>> On 1/19/2016 at 11:48 PM, in message
>> <22174.23240.402164.635740@mariner.uk.xensource.com>, Ian Jackson
>> <Ian.Jackson@eu.citrix.com> wrote:
>> > Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"):
>> > > Add pvusb APIs, including:
>> > >  - attach/detach (create/destroy) virtual usb controller.
>> > >  - attach/detach usb device
>> > >  - list usb controller and usb devices
>> > >  - some other helper functions
>> >
>> >
>> > Thanks.  This is making progress but I'm afraid we're not quite there
>> > yet.
>> >
>> >
>> > > +static int usbback_dev_unassign(libxl__gc *gc, const char *busid)
>> > > +{
>> > ...
>> > > +    /* Till here, USB device has been unbound from USBBACK and
>> > > +     * removed from xenstore, usb list couldn't show it anymore,
>> > > +     * so no matter removimg driver path successfully or not,
>> > > +     * we will report operation success.
>> > > +     */
>> >
>> > I'm still unconvinced by this and this may mean that the code in this
>> > function is in the wrong order.  Earlier we had this exchange:
>> >
>> > > > Ought this function to really report success if these calls fail ?
>> > >
>> > > I think so. Till here, the USB device has already been unbound from
>> > > usbback and removed from xenstore. usb-list cannot list it any more.
>> >
>> > The problem is that I think that if this function fails, it can leave
>> >  - debris in xenstore (the usbback path)
>> Yes, it's true.
>>
>> >  - the interface bound to the wrong driver
>> No, it won't be bound to 'wrong' driver, only maybe not bound to any driver
>> (Already unbound from usbback, but failed to rebound to its original
>> driver).
>> In this case, we would report warning: failed to rebind to driver xxx.
>>
>> > And then there is no way for the user to get libxl to re-attempt the
>> > operation, or clean up.  Am I right ?
>>
>> Yes. No way to re-attempt usbdev-detach or cleanup driver path in
>> xenstore. But won't affect next time usbdev-attach the same device.
>>
>> >
>> > One way to avoid this kind of problem is to deal with the xenstore
>> > path last.  That way the device will still appear as attached to the
>> > domain.
>>
>> I'm afraid if the side effect is acceptable? In my testing, some USB
>> bluetooth
>> device always fails to rebind to 'btusb' driver after it's unbound
>> from 'usbback'. In this case, we can't detach it from the domain then.
>
> Ian J., any opinion on this? If it's still thought to be better, I'll update patch.

I think Ian may be waiting for me to reply and express an opinion; but
unfortunately that will have to wait until next week. :-(

 -George

Re: [PATCH V13 3/5] libxl: add pvusb API

[Olaf Hering]

On Tue, Jan 19, Chunyan Liu wrote:

> +++ b/tools/libxl/libxl.c
> @@ -3204,7 +3204,7 @@ void libxl__device_disk_local_initiate_detach(libxl__egc *egc,
>          aodev->dev = device;
>          aodev->callback = local_device_detach_cb;
>          aodev->force = 0;
> -        libxl__initiate_device_remove(egc, aodev);
> +        libxl__initiate_device_generic_remove(egc, aodev);
>          return;
>      }
>  
> @@ -4172,8 +4172,10 @@ out:
>   * libxl_device_vkb_destroy
>   * libxl_device_vfb_remove
>   * libxl_device_vfb_destroy
> + * libxl_device_usbctrl_remove
> + * libxl_device_usbctrl_destroy

This should be moved down to DEFINE_DEVICE_REMOVE_CUSTOM.

>   */
> -#define DEFINE_DEVICE_REMOVE(type, removedestroy, f)                    \
> +#define DEFINE_DEVICE_REMOVE_EXT(type, remtype, removedestroy, f)        \
>      int libxl_device_##type##_##removedestroy(libxl_ctx *ctx,           \
>          uint32_t domid, libxl_device_##type *type,                      \
>          const libxl_asyncop_how *ao_how)                                \
> @@ -4193,13 +4195,19 @@ out:
>          aodev->dev = device;                                            \
>          aodev->callback = device_addrm_aocomplete;                      \
>          aodev->force = f;                                               \
> -        libxl__initiate_device_remove(egc, aodev);                      \
> +        libxl__initiate_device_##remtype##_remove(egc, aodev);          \
>                                                                          \
>      out:                                                                \
> -        if (rc) return AO_CREATE_FAIL(rc);                                    \
> +        if (rc) return AO_CREATE_FAIL(rc);                              \
>          return AO_INPROGRESS;                                           \
>      }
>  
> +#define DEFINE_DEVICE_REMOVE(type, removedestroy, f) \
> +    DEFINE_DEVICE_REMOVE_EXT(type, generic, removedestroy, f)
> +
> +#define DEFINE_DEVICE_REMOVE_CUSTOM(type, removedestroy, f)  \
> +    DEFINE_DEVICE_REMOVE_EXT(type, type, removedestroy, f)
> +
>  /* Define all remove/destroy functions and undef the macro */
>  
>  /* disk */


If this is the way to move forward, please split this out into a
separate change which can be applied to staging independent of any pvusb
changes.

I think the patch needs also the #undef.


> @@ -4223,6 +4231,10 @@ DEFINE_DEVICE_REMOVE(vfb, destroy, 1)
>  DEFINE_DEVICE_REMOVE(vtpm, remove, 0)
>  DEFINE_DEVICE_REMOVE(vtpm, destroy, 1)
>  
> +/* usbctrl */
> +DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, remove, 0)
> +DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, destroy, 1)
> +
>  /* channel/console hotunplug is not implemented. There are 2 possibilities:
>   * 1. add support for secondary consoles to xenconsoled
>   * 2. dynamically add/remove qemu chardevs via qmp messages. */

A comment should mention both libxl_device_usbctrl_remove/destroy and
libxl__initiate_device_usbctrl_remove/destroy.

Olaf

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 1/20/2016 at 12:56 PM, in message
<569F83F5020000660009E6AC@relay2.provo.novell.com>, "Chun Yan Liu"
<cyliu@suse.com> wrote: 

>  
>>>> On 1/19/2016 at 11:48 PM, in message 
> <22174.23240.402164.635740@mariner.uk.xensource.com>, Ian Jackson 
> <Ian.Jackson@eu.citrix.com> wrote:  
> > Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"):  
> > > Add pvusb APIs, including:  
> > >  - attach/detach (create/destroy) virtual usb controller.  
> > >  - attach/detach usb device  
> > >  - list usb controller and usb devices  
> > >  - some other helper functions  
> >   
> >   
> > Thanks.  This is making progress but I'm afraid we're not quite there  
> > yet.  
> >   
> >   
> > > +static int usbback_dev_unassign(libxl__gc *gc, const char *busid)  
> > > +{  
> > ...  
> > > +    /* Till here, USB device has been unbound from USBBACK and  
> > > +     * removed from xenstore, usb list couldn't show it anymore,  
> > > +     * so no matter removimg driver path successfully or not,  
> > > +     * we will report operation success.  
> > > +     */  
> >   
> > I'm still unconvinced by this and this may mean that the code in this  
> > function is in the wrong order.  Earlier we had this exchange:  
> >   
> > > > Ought this function to really report success if these calls fail ?  
> > >   
> > > I think so. Till here, the USB device has already been unbound from  
> > > usbback and removed from xenstore. usb-list cannot list it any more.  
> >   
> > The problem is that I think that if this function fails, it can leave  
> >  - debris in xenstore (the usbback path)  
> Yes, it's true. 
>  
> >  - the interface bound to the wrong driver 
> No, it won't be bound to 'wrong' driver, only maybe not bound to any driver 
> (Already unbound from usbback, but failed to rebound to its original  
> driver). 
> In this case, we would report warning: failed to rebind to driver xxx.  
>  
> > And then there is no way for the user to get libxl to re-attempt the  
> > operation, or clean up.  Am I right ? 
>  
> Yes. No way to re-attempt usbdev-detach or cleanup driver path in 
> xenstore. But won't affect next time usbdev-attach the same device. 
>   
> >   
> > One way to avoid this kind of problem is to deal with the xenstore  
> > path last.  That way the device will still appear as attached to the  
> > domain.  
>  
> I'm afraid if the side effect is acceptable? In my testing, some USB  
> bluetooth 
> device always fails to rebind to 'btusb' driver after it's unbound 
> from 'usbback'. In this case, we can't detach it from the domain then.  

Ian J., any opinion on this? If it's still thought to be better, I'll update patch.

>  
> >   
> > To do this properly I think bind_usbintf may need to become idempotent  
> > even in the face of other callers racing to assign the device.  I  
> > think that would mean bind_usbif it would have to know what driver to  
> > expect to find the device bound to. 

bind_usbintf already has parameter to indicate which driver to be bound to.

- Chunyan
> >   
> > George, am I right here ?  
> >   
> >   
> > I have a few other comments too:  
> >   
> > > +/* get original driver path of usb interface, stored in @drvpath */  
> > > +static int usbintf_get_drvpath(libxl__gc *gc, const char *intf, char   
> > **drvpath)  
> > > +{  
> > > +    char *spath, *dp = NULL;  
> > > +    struct stat st;  
> > > +    int rc;  
> > > +  
> > > +    spath = GCSPRINTF(SYSFS_USB_DEV "/%s/driver", intf);  
> > > +  
> > > +    rc = lstat(spath, &st);  
> >   
> > This `rc' should be `r'.  (CODING_STYLE.)  
> >   
> > I mentioned this in my review of v12 in the context of  
> > sysfs_write_intf.  But there is still more than one occurrence in v13  
> > of `rc' for a syscall return value.  
> >   
>  
> Sorry, will double check again. 
>  
> >   
> > > +static int sysfs_write_intf(libxl__gc *gc, const char *intf, const char   
> > *path)  
> > > +{  
> >   
> > Last time I wrote:  
> >   
> >   But there is nothing usb specific about this function.  Looking for  
> >   similar code elsewhere this function seems very similar to  
> >   libxl_pci.c:sysfs_write_bdf, but I won't ask you to try to refactor  
> >   these two functions together.  
> >   
> > This time I have remembered that libxl_write_exactly exists, and could  
> > be used.  Sorry for not saing this last time but I think you can  
> > probably just get rid of this in favour of libxl_write_exactly.  If  
> > you think not, please say why.  
>  
> After checking the codes, yes, except for open and close fd, 
> libxl_write_exactly does the work. Will reuse it. 
>  
> >   
> >   
> > > +static int bind_usbintf(libxl__gc *gc, const char *intf, const char   
> > *drvpath)  
> > > +{  
> > > +    char *path;  
> > > +    struct stat st;  
> > > +  
> > > +    path = GCSPRINTF("%s/%s", drvpath, intf);  
> > > +    /* if already bound, return */  
> > > +    if (!lstat(path, &st))  
> > > +        return 0;  
> > > +    else if (errno != ENOENT)  
> > > +        return ERROR_FAIL;  
> >   
> > This new ERROR_FAIL fails to log anything, and probably should.  I  
> > think the anomalous style leads to this mistake.  You should say  
> >        r = lstat...  
> > and adopt the pattern from the rest of libxl.  
>  
> Will update. 
>  
> Thanks, 
> Chunyan 
>  
> >   
> >   
> > Thanks,  
> > Ian.  
> >   
> >   
>  
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[Chun Yan Liu]

>>> On 1/19/2016 at 11:48 PM, in message
<22174.23240.402164.635740@mariner.uk.xensource.com>, Ian Jackson
<Ian.Jackson@eu.citrix.com> wrote: 
> Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"): 
> > Add pvusb APIs, including: 
> >  - attach/detach (create/destroy) virtual usb controller. 
> >  - attach/detach usb device 
> >  - list usb controller and usb devices 
> >  - some other helper functions 
>  
>  
> Thanks.  This is making progress but I'm afraid we're not quite there 
> yet. 
>  
>  
> > +static int usbback_dev_unassign(libxl__gc *gc, const char *busid) 
> > +{ 
> ... 
> > +    /* Till here, USB device has been unbound from USBBACK and 
> > +     * removed from xenstore, usb list couldn't show it anymore, 
> > +     * so no matter removimg driver path successfully or not, 
> > +     * we will report operation success. 
> > +     */ 
>  
> I'm still unconvinced by this and this may mean that the code in this 
> function is in the wrong order.  Earlier we had this exchange: 
>  
> > > Ought this function to really report success if these calls fail ? 
> >  
> > I think so. Till here, the USB device has already been unbound from 
> > usbback and removed from xenstore. usb-list cannot list it any more. 
>  
> The problem is that I think that if this function fails, it can leave 
>  - debris in xenstore (the usbback path) 
Yes, it's true.

>  - the interface bound to the wrong driver
No, it won't be bound to 'wrong' driver, only maybe not bound to any driver
(Already unbound from usbback, but failed to rebound to its original driver).
In this case, we would report warning: failed to rebind to driver xxx. 

> And then there is no way for the user to get libxl to re-attempt the 
> operation, or clean up.  Am I right ?

Yes. No way to re-attempt usbdev-detach or cleanup driver path in
xenstore. But won't affect next time usbdev-attach the same device.
 
>  
> One way to avoid this kind of problem is to deal with the xenstore 
> path last.  That way the device will still appear as attached to the 
> domain. 

I'm afraid if the side effect is acceptable. In my testing, some USB bluetooth
device always fails to rebind to 'btusb' driver after it's unbound
from 'usbback'. In this case, we can't detach it from the domain then. 

>  
> To do this properly I think bind_usbintf may need to become idempotent 
> even in the face of other callers racing to assign the device.  I 
> think that would mean bind_usbif it would have to know what driver to 
> expect to find the device bound to. 
>  
> George, am I right here ? 
>  
>  
> I have a few other comments too: 
>  
> > +/* get original driver path of usb interface, stored in @drvpath */ 
> > +static int usbintf_get_drvpath(libxl__gc *gc, const char *intf, char  
> **drvpath) 
> > +{ 
> > +    char *spath, *dp = NULL; 
> > +    struct stat st; 
> > +    int rc; 
> > + 
> > +    spath = GCSPRINTF(SYSFS_USB_DEV "/%s/driver", intf); 
> > + 
> > +    rc = lstat(spath, &st); 
>  
> This `rc' should be `r'.  (CODING_STYLE.) 
>  
> I mentioned this in my review of v12 in the context of 
> sysfs_write_intf.  But there is still more than one occurrence in v13 
> of `rc' for a syscall return value. 
>  

Sorry, will double check again.

>  
> > +static int sysfs_write_intf(libxl__gc *gc, const char *intf, const char  
> *path) 
> > +{ 
>  
> Last time I wrote: 
>  
>   But there is nothing usb specific about this function.  Looking for 
>   similar code elsewhere this function seems very similar to 
>   libxl_pci.c:sysfs_write_bdf, but I won't ask you to try to refactor 
>   these two functions together. 
>  
> This time I have remembered that libxl_write_exactly exists, and could 
> be used.  Sorry for not saing this last time but I think you can 
> probably just get rid of this in favour of libxl_write_exactly.  If 
> you think not, please say why. 

After checking the codes, yes, except for open and close fd,
libxl_write_exactly does the work. Will reuse it.

>  
>  
> > +static int bind_usbintf(libxl__gc *gc, const char *intf, const char  
> *drvpath) 
> > +{ 
> > +    char *path; 
> > +    struct stat st; 
> > + 
> > +    path = GCSPRINTF("%s/%s", drvpath, intf); 
> > +    /* if already bound, return */ 
> > +    if (!lstat(path, &st)) 
> > +        return 0; 
> > +    else if (errno != ENOENT) 
> > +        return ERROR_FAIL; 
>  
> This new ERROR_FAIL fails to log anything, and probably should.  I 
> think the anomalous style leads to this mistake.  You should say 
>        r = lstat... 
> and adopt the pattern from the rest of libxl. 

Will update.

Thanks,
Chunyan

>  
>  
> Thanks, 
> Ian. 
>  
>  

Re: [PATCH V13 3/5] libxl: add pvusb API

[Ian Jackson]

Chunyan Liu writes ("[PATCH V13 3/5] libxl: add pvusb API"):
> Add pvusb APIs, including:
>  - attach/detach (create/destroy) virtual usb controller.
>  - attach/detach usb device
>  - list usb controller and usb devices
>  - some other helper functions


Thanks.  This is making progress but I'm afraid we're not quite there
yet.


> +static int usbback_dev_unassign(libxl__gc *gc, const char *busid)
> +{
...
> +    /* Till here, USB device has been unbound from USBBACK and
> +     * removed from xenstore, usb list couldn't show it anymore,
> +     * so no matter removimg driver path successfully or not,
> +     * we will report operation success.
> +     */

I'm still unconvinced by this and this may mean that the code in this
function is in the wrong order.  Earlier we had this exchange:

> > Ought this function to really report success if these calls fail ?
> 
> I think so. Till here, the USB device has already been unbound from
> usbback and removed from xenstore. usb-list cannot list it any more.

The problem is that I think that if this function fails, it can leave
 - debris in xenstore (the usbback path)
 - the interface bound to the wrong driver
And then there is no way for the user to get libxl to re-attempt the
operation, or clean up.  Am I right ?

One way to avoid this kind of problem is to deal with the xenstore
path last.  That way the device will still appear as attached to the
domain.

To do this properly I think bind_usbintf may need to become idempotent
even in the face of other callers racing to assign the device.  I
think that would mean bind_usbif it would have to know what driver to
expect to find the device bound to.

George, am I right here ?


I have a few other comments too:

> +/* get original driver path of usb interface, stored in @drvpath */
> +static int usbintf_get_drvpath(libxl__gc *gc, const char *intf, char **drvpath)
> +{
> +    char *spath, *dp = NULL;
> +    struct stat st;
> +    int rc;
> +
> +    spath = GCSPRINTF(SYSFS_USB_DEV "/%s/driver", intf);
> +
> +    rc = lstat(spath, &st);

This `rc' should be `r'.  (CODING_STYLE.)

I mentioned this in my review of v12 in the context of
sysfs_write_intf.  But there is still more than one occurrence in v13
of `rc' for a syscall return value.


> +static int sysfs_write_intf(libxl__gc *gc, const char *intf, const char *path)
> +{

Last time I wrote:

  But there is nothing usb specific about this function.  Looking for
  similar code elsewhere this function seems very similar to
  libxl_pci.c:sysfs_write_bdf, but I won't ask you to try to refactor
  these two functions together.

This time I have remembered that libxl_write_exactly exists, and could
be used.  Sorry for not saing this last time but I think you can
probably just get rid of this in favour of libxl_write_exactly.  If
you think not, please say why.


> +static int bind_usbintf(libxl__gc *gc, const char *intf, const char *drvpath)
> +{
> +    char *path;
> +    struct stat st;
> +
> +    path = GCSPRINTF("%s/%s", drvpath, intf);
> +    /* if already bound, return */
> +    if (!lstat(path, &st))
> +        return 0;
> +    else if (errno != ENOENT)
> +        return ERROR_FAIL;

This new ERROR_FAIL fails to log anything, and probably should.  I
think the anomalous style leads to this mistake.  You should say
       r = lstat...
and adopt the pattern from the rest of libxl.


Thanks,
Ian.

[PATCH V13 3/5] libxl: add pvusb API

[Chunyan Liu]

Add pvusb APIs, including:
 - attach/detach (create/destroy) virtual usb controller.
 - attach/detach usb device
 - list usb controller and usb devices
 - some other helper functions

Signed-off-by: Chunyan Liu <cyliu@suse.com>
Signed-off-by: Simon Cao <caobosimon@gmail.com>
Signed-off-by: George Dunlap <george.dunlap@citrix.com>
---
changes:
* address error handlings

 tools/libxl/Makefile                 |    2 +-
 tools/libxl/libxl.c                  |   34 +-
 tools/libxl/libxl.h                  |   77 ++
 tools/libxl/libxl_device.c           |   13 +-
 tools/libxl/libxl_internal.h         |   22 +-
 tools/libxl/libxl_osdeps.h           |   13 +
 tools/libxl/libxl_pvusb.c            | 1567 ++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_types.idl          |   46 +
 tools/libxl/libxl_types_internal.idl |    1 +
 tools/libxl/libxl_utils.c            |   18 +
 tools/libxl/libxl_utils.h            |    5 +
 11 files changed, 1785 insertions(+), 13 deletions(-)
 create mode 100644 tools/libxl/libxl_pvusb.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 2abae0c..e25ffa6 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -104,7 +104,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
 			libxl_stream_read.o libxl_stream_write.o \
 			libxl_save_callout.o _libxl_save_msgs_callout.o \
 			libxl_qmp.o libxl_event.o libxl_fork.o \
-			libxl_dom_suspend.o $(LIBXL_OBJS-y)
+			libxl_dom_suspend.o libxl_pvusb.o $(LIBXL_OBJS-y)
 LIBXL_OBJS += libxl_genid.o
 LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
 
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 43d5709..920c135 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -3204,7 +3204,7 @@ void libxl__device_disk_local_initiate_detach(libxl__egc *egc,
         aodev->dev = device;
         aodev->callback = local_device_detach_cb;
         aodev->force = 0;
-        libxl__initiate_device_remove(egc, aodev);
+        libxl__initiate_device_generic_remove(egc, aodev);
         return;
     }
 
@@ -4172,8 +4172,10 @@ out:
  * libxl_device_vkb_destroy
  * libxl_device_vfb_remove
  * libxl_device_vfb_destroy
+ * libxl_device_usbctrl_remove
+ * libxl_device_usbctrl_destroy
  */
-#define DEFINE_DEVICE_REMOVE(type, removedestroy, f)                    \
+#define DEFINE_DEVICE_REMOVE_EXT(type, remtype, removedestroy, f)        \
     int libxl_device_##type##_##removedestroy(libxl_ctx *ctx,           \
         uint32_t domid, libxl_device_##type *type,                      \
         const libxl_asyncop_how *ao_how)                                \
@@ -4193,13 +4195,19 @@ out:
         aodev->dev = device;                                            \
         aodev->callback = device_addrm_aocomplete;                      \
         aodev->force = f;                                               \
-        libxl__initiate_device_remove(egc, aodev);                      \
+        libxl__initiate_device_##remtype##_remove(egc, aodev);          \
                                                                         \
     out:                                                                \
-        if (rc) return AO_CREATE_FAIL(rc);                                    \
+        if (rc) return AO_CREATE_FAIL(rc);                              \
         return AO_INPROGRESS;                                           \
     }
 
+#define DEFINE_DEVICE_REMOVE(type, removedestroy, f) \
+    DEFINE_DEVICE_REMOVE_EXT(type, generic, removedestroy, f)
+
+#define DEFINE_DEVICE_REMOVE_CUSTOM(type, removedestroy, f)  \
+    DEFINE_DEVICE_REMOVE_EXT(type, type, removedestroy, f)
+
 /* Define all remove/destroy functions and undef the macro */
 
 /* disk */
@@ -4223,6 +4231,10 @@ DEFINE_DEVICE_REMOVE(vfb, destroy, 1)
 DEFINE_DEVICE_REMOVE(vtpm, remove, 0)
 DEFINE_DEVICE_REMOVE(vtpm, destroy, 1)
 
+/* usbctrl */
+DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, remove, 0)
+DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, destroy, 1)
+
 /* channel/console hotunplug is not implemented. There are 2 possibilities:
  * 1. add support for secondary consoles to xenconsoled
  * 2. dynamically add/remove qemu chardevs via qmp messages. */
@@ -4236,6 +4248,8 @@ DEFINE_DEVICE_REMOVE(vtpm, destroy, 1)
  * libxl_device_disk_add
  * libxl_device_nic_add
  * libxl_device_vtpm_add
+ * libxl_device_usbctrl_add
+ * libxl_device_usbdev_add
  */
 
 #define DEFINE_DEVICE_ADD(type)                                         \
@@ -4267,6 +4281,12 @@ DEFINE_DEVICE_ADD(nic)
 /* vtpm */
 DEFINE_DEVICE_ADD(vtpm)
 
+/* usbctrl */
+DEFINE_DEVICE_ADD(usbctrl)
+
+/* usb */
+DEFINE_DEVICE_ADD(usbdev)
+
 #undef DEFINE_DEVICE_ADD
 
 /******************************************************************************/
@@ -4432,7 +4452,7 @@ static int remove_device(libxl__egc *egc, libxl__ao *ao,
         aodev->dev = dev;
         aodev->action = LIBXL__DEVICE_ACTION_REMOVE;
         aodev->callback = device_complete;
-        libxl__initiate_device_remove(egc, aodev);
+        libxl__initiate_device_generic_remove(egc, aodev);
         break;
     case LIBXL__DEVICE_KIND_QDISK:
         if (--dguest->num_qdisks == 0) {
@@ -6807,6 +6827,10 @@ int libxl_retrieve_domain_configuration(libxl_ctx *ctx, uint32_t domid,
 
     MERGE(pci, pcidevs, COMPARE_PCI, {});
 
+    MERGE(usbctrl, usbctrls, COMPARE_USBCTRL, {});
+
+    MERGE(usbdev, usbdevs, COMPARE_USB, {});
+
     /* Take care of removable device. We maintain invariant in the
      * insert / remove operation so that:
      * 1. if xenstore is "empty" while JSON is not, the result
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 0e347b9..c708cc8 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -123,6 +123,12 @@
 #define LIBXL_HAVE_DOMAIN_NODEAFFINITY 1
 
 /*
+ * LIBXL_HAVE_PVUSB indicates functions for plugging in USB devices
+ * through pvusb -- both hotplug and at domain creation time..
+ */
+#define LIBXL_HAVE_PVUSB 1
+
+/*
  * LIBXL_HAVE_BUILDINFO_HVM_VENDOR_DEVICE indicates that the
  * libxl_vendor_device field is present in the hvm sections of
  * libxl_domain_build_info. This field tells libxl which
@@ -1503,6 +1509,77 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk,
                        const libxl_asyncop_how *ao_how)
                        LIBXL_EXTERNAL_CALLERS_ONLY;
 
+/*
+ * USB
+ *
+ * For each device removed or added, one of these protocols is available:
+ * - PV (i.e., PVUSB)
+ * - DEVICEMODEL (i.e, qemu)
+ *
+ * PV is available for either PV or HVM domains.  DEVICEMODEL is only
+ * available for HVM domains.  The caller can additionally specify
+ * "AUTO", in which case the library will try to determine the best
+ * protocol automatically.
+ *
+ * At the moment, the only protocol implemented is PV.
+ *
+ * One can add/remove USB controllers to/from guest, and attach/detach USB
+ * devices to/from USB controllers.
+ *
+ * To add USB controllers and USB devices, one can adding USB controllers
+ * first and then attaching USB devices to some USB controller, or adding
+ * USB devices to guest directly, it will automatically create a USB
+ * controller for USB devices to attach.
+ *
+ * To remove USB controllers or USB devices, one can remove USB devices
+ * under USB controller one by one and then remove USB controller, or
+ * remove USB controller directly, it will remove all USB devices under
+ * it automatically.
+ *
+ */
+/* USB Controllers*/
+int libxl_device_usbctrl_add(libxl_ctx *ctx, uint32_t domid,
+                             libxl_device_usbctrl *usbctrl,
+                             const libxl_asyncop_how *ao_how)
+                             LIBXL_EXTERNAL_CALLERS_ONLY;
+
+int libxl_device_usbctrl_remove(libxl_ctx *ctx, uint32_t domid,
+                                libxl_device_usbctrl *usbctrl,
+                                const libxl_asyncop_how *ao_how)
+                                LIBXL_EXTERNAL_CALLERS_ONLY;
+
+int libxl_device_usbctrl_destroy(libxl_ctx *ctx, uint32_t domid,
+                                 libxl_device_usbctrl *usbctrl,
+                                 const libxl_asyncop_how *ao_how)
+                                 LIBXL_EXTERNAL_CALLERS_ONLY;
+
+libxl_device_usbctrl *libxl_device_usbctrl_list(libxl_ctx *ctx,
+                                                uint32_t domid, int *num);
+
+void libxl_device_usbctrl_list_free(libxl_device_usbctrl *list, int nr);
+
+
+int libxl_device_usbctrl_getinfo(libxl_ctx *ctx, uint32_t domid,
+                                 libxl_device_usbctrl *usbctrl,
+                                 libxl_usbctrlinfo *usbctrlinfo);
+
+/* USB Devices */
+
+int libxl_device_usbdev_add(libxl_ctx *ctx, uint32_t domid,
+                            libxl_device_usbdev *usbdev,
+                            const libxl_asyncop_how *ao_how)
+                            LIBXL_EXTERNAL_CALLERS_ONLY;
+
+int libxl_device_usbdev_remove(libxl_ctx *ctx, uint32_t domid,
+                               libxl_device_usbdev *usbdev,
+                               const libxl_asyncop_how *ao_how)
+                               LIBXL_EXTERNAL_CALLERS_ONLY;
+
+libxl_device_usbdev *
+libxl_device_usbdev_list(libxl_ctx *ctx, uint32_t domid, int *num);
+
+void libxl_device_usbdev_list_free(libxl_device_usbdev *list, int nr);
+
 /* Network Interfaces */
 int libxl_device_nic_add(libxl_ctx *ctx, uint32_t domid, libxl_device_nic *nic,
                          const libxl_asyncop_how *ao_how)
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index 8bb5e93..b7a6a13 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -676,7 +676,10 @@ void libxl__devices_destroy(libxl__egc *egc, libxl__devices_remove_state *drs)
                 aodev->action = LIBXL__DEVICE_ACTION_REMOVE;
                 aodev->dev = dev;
                 aodev->force = drs->force;
-                libxl__initiate_device_remove(egc, aodev);
+                if (dev->backend_kind == LIBXL__DEVICE_KIND_VUSB)
+                    libxl__initiate_device_usbctrl_remove(egc, aodev);
+                else
+                    libxl__initiate_device_generic_remove(egc, aodev);
             }
         }
     }
@@ -775,8 +778,8 @@ out:
     return;
 }
 
-void libxl__initiate_device_remove(libxl__egc *egc,
-                                   libxl__ao_device *aodev)
+void libxl__initiate_device_generic_remove(libxl__egc *egc,
+                                           libxl__ao_device *aodev)
 {
     STATE_AO_GC(aodev->ao);
     xs_transaction_t t = 0;
@@ -806,7 +809,7 @@ void libxl__initiate_device_remove(libxl__egc *egc,
             (info.paused || info.dying || info.shutdown)) {
             /*
              * TODO: 4.2 Bodge due to QEMU, see comment on top of
-             * libxl__initiate_device_remove in libxl_internal.h
+             * libxl__initiate_device_generic_remove in libxl_internal.h
              */
             rc = libxl__ev_time_register_rel(ao, &aodev->timeout,
                                              device_qemu_timeout,
@@ -942,7 +945,7 @@ static void device_backend_callback(libxl__egc *egc, libxl__ev_devstate *ds,
         !aodev->force) {
         LOG(DEBUG, "Timeout reached, initiating forced remove");
         aodev->force = 1;
-        libxl__initiate_device_remove(egc, aodev);
+        libxl__initiate_device_generic_remove(egc, aodev);
         return;
     }
 
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index d1eb18f..0ccad9a 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2583,6 +2583,14 @@ _hidden void libxl__device_vtpm_add(libxl__egc *egc, uint32_t domid,
                                    libxl_device_vtpm *vtpm,
                                    libxl__ao_device *aodev);
 
+_hidden void libxl__device_usbctrl_add(libxl__egc *egc, uint32_t domid,
+                                       libxl_device_usbctrl *usbctrl,
+                                       libxl__ao_device *aodev);
+
+_hidden void libxl__device_usbdev_add(libxl__egc *egc, uint32_t domid,
+                                      libxl_device_usbdev *usbdev,
+                                      libxl__ao_device *aodev);
+
 /* Internal function to connect a vkb device */
 _hidden int libxl__device_vkb_add(libxl__gc *gc, uint32_t domid,
                                   libxl_device_vkb *vkb);
@@ -2612,8 +2620,15 @@ _hidden void libxl__wait_device_connection(libxl__egc*,
  *
  * Once finished, aodev->callback will be executed.
  */
-_hidden void libxl__initiate_device_remove(libxl__egc *egc,
-                                           libxl__ao_device *aodev);
+_hidden void libxl__initiate_device_generic_remove(libxl__egc *egc,
+                                                   libxl__ao_device *aodev);
+
+_hidden int libxl__device_from_usbctrl(libxl__gc *gc, uint32_t domid,
+                               libxl_device_usbctrl *usbctrl,
+                               libxl__device *device);
+
+_hidden void libxl__initiate_device_usbctrl_remove(libxl__egc *egc,
+                                                   libxl__ao_device *aodev);
 
 /*
  * libxl__get_hotplug_script_info returns the args and env that should
@@ -3975,6 +3990,9 @@ static inline void libxl__update_config_vtpm(libxl__gc *gc,
 #define COMPARE_PCI(a, b) ((a)->func == (b)->func &&    \
                            (a)->bus == (b)->bus &&      \
                            (a)->dev == (b)->dev)
+#define COMPARE_USB(a, b) ((a)->ctrl == (b)->ctrl && \
+                           (a)->port == (b)->port)
+#define COMPARE_USBCTRL(a, b) ((a)->devid == (b)->devid)
 
 /* DEVICE_ADD
  *
diff --git a/tools/libxl/libxl_osdeps.h b/tools/libxl/libxl_osdeps.h
index d9661c9..802c762 100644
--- a/tools/libxl/libxl_osdeps.h
+++ b/tools/libxl/libxl_osdeps.h
@@ -24,6 +24,8 @@
 #define _GNU_SOURCE
 
 #if defined(__NetBSD__)
+#define SYSFS_USB_DEV          "/sys/bus/usb/devices"
+#define SYSFS_USBBACK_DRIVER   "/kern/xen/usb"
 #define SYSFS_PCI_DEV          "/sys/bus/pci/devices"
 #define SYSFS_PCIBACK_DRIVER   "/kern/xen/pci"
 #define NETBACK_NIC_NAME       "xvif%ui%d"
@@ -31,6 +33,8 @@
 #elif defined(__OpenBSD__)
 #include <util.h>
 #elif defined(__linux__)
+#define SYSFS_USB_DEV          "/sys/bus/usb/devices"
+#define SYSFS_USBBACK_DRIVER   "/sys/bus/usb/drivers/usbback"
 #define SYSFS_PCI_DEV          "/sys/bus/pci/devices"
 #define SYSFS_PCIBACK_DRIVER   "/sys/bus/pci/drivers/pciback"
 #define NETBACK_NIC_NAME       "vif%u.%d"
@@ -38,6 +42,8 @@
 #elif defined(__sun__)
 #include <stropts.h>
 #elif defined(__FreeBSD__)
+#define SYSFS_USB_DEV          "/dev/null"
+#define SYSFS_USBBACK_DRIVER   "/dev/null"
 #define SYSFS_PCI_DEV          "/dev/null"
 #define SYSFS_PCIBACK_DRIVER   "/dev/null"
 #define NETBACK_NIC_NAME       "xnb%u.%d"
@@ -45,6 +51,13 @@
 #include <sys/endian.h>
 #endif
 
+#ifndef SYSFS_USBBACK_DRIVER
+#error define SYSFS_USBBACK_DRIVER for your platform
+#endif
+#ifndef SYSFS_USB_DEV
+#error define SYSFS_USB_DEV for your platform
+#endif
+
 #ifndef SYSFS_PCIBACK_DRIVER
 #error define SYSFS_PCIBACK_DRIVER for your platform
 #endif
diff --git a/tools/libxl/libxl_pvusb.c b/tools/libxl/libxl_pvusb.c
new file mode 100644
index 0000000..1869611
--- /dev/null
+++ b/tools/libxl/libxl_pvusb.c
@@ -0,0 +1,1567 @@
+/*
+ * Copyright (C) 2015 SUSE LINUX Products GmbH, Nuernberg, Germany.
+ * Author Chunyan Liu <cyliu@suse.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+#include <inttypes.h>
+
+#define USBBACK_INFO_PATH "/libxl/usbback"
+
+#define USBHUB_CLASS_CODE 9
+
+static int libxl__device_usbctrl_setdefault(libxl__gc *gc, uint32_t domid,
+                                            libxl_device_usbctrl *usbctrl)
+{
+    int rc;
+    libxl_domain_type domtype = libxl__domain_type(gc, domid);
+
+    if (!usbctrl->version)
+        usbctrl->version = 2;
+
+    if (!usbctrl->ports)
+        usbctrl->ports = 8;
+
+    if (usbctrl->type == LIBXL_USBCTRL_TYPE_AUTO) {
+        if (domtype == LIBXL_DOMAIN_TYPE_PV) {
+            usbctrl->type = LIBXL_USBCTRL_TYPE_PV;
+        } else if (domtype == LIBXL_DOMAIN_TYPE_HVM) {
+            /* FIXME: See if we can detect PV frontend */
+            usbctrl->type = LIBXL_USBCTRL_TYPE_DEVICEMODEL;
+        }
+    }
+
+    rc = libxl__resolve_domid(gc, usbctrl->backend_domname,
+                              &usbctrl->backend_domid);
+    return rc;
+}
+
+int libxl__device_from_usbctrl(libxl__gc *gc, uint32_t domid,
+                               libxl_device_usbctrl *usbctrl,
+                               libxl__device *device)
+{
+    device->backend_devid   = usbctrl->devid;
+    device->backend_domid   = usbctrl->backend_domid;
+    device->backend_kind    = LIBXL__DEVICE_KIND_VUSB;
+    device->devid           = usbctrl->devid;
+    device->domid           = domid;
+    device->kind            = LIBXL__DEVICE_KIND_VUSB;
+
+    return 0;
+}
+
+/* Add usbctrl information to xenstore.
+ *
+ * Adding a usb controller will add a new 'vusb' device in xenstore, and
+ * add corresponding frontend, backend information to it. According to
+ * "update_json", decide wether to update json config file.
+ */
+static int libxl__device_usbctrl_add_xenstore(libxl__gc *gc, uint32_t domid,
+                                              libxl_device_usbctrl *usbctrl,
+                                              bool update_json)
+{
+    libxl__device *device;
+    flexarray_t *front;
+    flexarray_t *back;
+    xs_transaction_t t = XBT_NULL;
+    int i, rc;
+    libxl_domain_config d_config;
+    libxl_device_usbctrl usbctrl_saved;
+    libxl__domain_userdata_lock *lock = NULL;
+
+    libxl_domain_config_init(&d_config);
+    libxl_device_usbctrl_init(&usbctrl_saved);
+    libxl_device_usbctrl_copy(CTX, &usbctrl_saved, usbctrl);
+
+    GCNEW(device);
+    rc = libxl__device_from_usbctrl(gc, domid, usbctrl, device);
+    if (rc) goto out;
+
+    front = flexarray_make(gc, 4, 1);
+    back = flexarray_make(gc, 12, 1);
+
+    flexarray_append_pair(back, "frontend-id", GCSPRINTF("%d", domid));
+    flexarray_append_pair(back, "online", "1");
+    flexarray_append_pair(back, "state",
+                          GCSPRINTF("%d", XenbusStateInitialising));
+    flexarray_append_pair(back, "type",
+                          (char *)libxl_usbctrl_type_to_string(usbctrl->type));
+    flexarray_append_pair(back, "usb-ver", GCSPRINTF("%d", usbctrl->version));
+    flexarray_append_pair(back, "num-ports", GCSPRINTF("%d", usbctrl->ports));
+    flexarray_append_pair(back, "port", "");
+    for (i = 0; i < usbctrl->ports; i++)
+        flexarray_append_pair(back, GCSPRINTF("port/%d", i + 1), "");
+
+    flexarray_append_pair(front, "backend-id",
+                          GCSPRINTF("%d", usbctrl->backend_domid));
+    flexarray_append_pair(front, "state",
+                          GCSPRINTF("%d", XenbusStateInitialising));
+
+    if (update_json) {
+        lock = libxl__lock_domain_userdata(gc, domid);
+        if (!lock) {
+            rc = ERROR_LOCK_FAIL;
+            goto out;
+        }
+
+        rc = libxl__get_domain_configuration(gc, domid, &d_config);
+        if (rc) goto out;
+
+        DEVICE_ADD(usbctrl, usbctrls, domid, &usbctrl_saved,
+                   COMPARE_USBCTRL, &d_config);
+    }
+
+    for (;;) {
+        rc = libxl__xs_transaction_start(gc, &t);
+        if (rc) goto out;
+
+        rc = libxl__device_exists(gc, t, device);
+        if (rc < 0) goto out;
+        if (rc == 1) {
+            /* already exists in xenstore */
+            LOG(ERROR, "device already exists in xenstore");
+            rc = ERROR_DEVICE_EXISTS;
+            goto out;
+        }
+
+        if (update_json) {
+            rc = libxl__set_domain_configuration(gc, domid, &d_config);
+            if (rc) goto out;
+        }
+
+        libxl__device_generic_add(gc, t, device,
+                          libxl__xs_kvs_of_flexarray(gc, back, back->count),
+                          libxl__xs_kvs_of_flexarray(gc, front, front->count),
+                          NULL);
+
+        rc = libxl__xs_transaction_commit(gc, &t);
+        if (!rc) break;
+        if (rc < 0) goto out;
+    }
+
+out:
+    libxl__xs_transaction_abort(gc, &t);
+    if (lock) libxl__unlock_domain_userdata(lock);
+    libxl_device_usbctrl_dispose(&usbctrl_saved);
+    libxl_domain_config_dispose(&d_config);
+    return rc;
+}
+
+/* AO operation to add a usb controller.
+ *
+ * Generally, it does:
+ * 1) fill in necessary usb controler information with default value
+ * 2) write usb controller frontend/backend info to xenstore, update json
+ *    config file if necessary.
+ * 3) wait for device connection. PVUSB frontend and backend driver will
+ *    probe xenstore paths and build connection between frontend and backend.
+ *
+ * Before calling this function, aodev should be properly filled:
+ * aodev->ao, aodev->callback, aodev->update_json, ...
+ */
+void libxl__device_usbctrl_add(libxl__egc *egc, uint32_t domid,
+                               libxl_device_usbctrl *usbctrl,
+                               libxl__ao_device *aodev)
+{
+    STATE_AO_GC(aodev->ao);
+    libxl__device *device;
+    int rc;
+
+    rc = libxl__device_usbctrl_setdefault(gc, domid, usbctrl);
+    if (rc < 0) goto out;
+
+    if (usbctrl->devid == -1) {
+        usbctrl->devid = libxl__device_nextid(gc, domid, "vusb");
+        if (usbctrl->devid < 0) {
+            rc = ERROR_FAIL;
+            goto out;
+        }
+    }
+
+    if (usbctrl->type != LIBXL_USBCTRL_TYPE_PV) {
+        LOG(ERROR, "Unsupported USB controller type");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = libxl__device_usbctrl_add_xenstore(gc, domid, usbctrl,
+                                            aodev->update_json);
+    if (rc) goto out;
+
+    GCNEW(device);
+    rc = libxl__device_from_usbctrl(gc, domid, usbctrl, device);
+    if (rc) goto out;
+
+    aodev->dev = device;
+    aodev->action = LIBXL__DEVICE_ACTION_ADD;
+    libxl__wait_device_connection(egc, aodev);
+    return;
+
+out:
+    aodev->rc = rc;
+    aodev->callback(egc, aodev);
+    return;
+}
+
+static int libxl__device_usbdev_list_for_usbctrl(libxl__gc *gc, uint32_t domid,
+                                                 libxl_devid usbctrl,
+                                                 libxl_device_usbdev **usbdevs,
+                                                 int *num);
+
+static int libxl__device_usbdev_remove(libxl__gc *gc, uint32_t domid,
+                                       libxl_device_usbdev *usbdev);
+
+/* AO function to remove a usb controller.
+ *
+ * Generally, it does:
+ * 1) check if the usb controller exists or not
+ * 2) remove all usb devices under controller
+ * 3) remove usb controller information from xenstore
+ *
+ * Before calling this function, aodev should be properly filled:
+ * aodev->ao, aodev->dev, aodev->callback, ...
+ */
+void libxl__initiate_device_usbctrl_remove(libxl__egc *egc,
+                                           libxl__ao_device *aodev)
+{
+    STATE_AO_GC(aodev->ao);
+    libxl_device_usbdev *usbdevs = NULL;
+    int num_usbdev = 0;
+    int i, rc;
+    uint32_t domid = ao->domid;
+    int usbctrl_devid = aodev->dev->devid;
+    libxl_device_usbctrl usbctrl;
+    libxl_usbctrlinfo usbctrlinfo;
+
+    libxl_device_usbctrl_init(&usbctrl);
+    libxl_usbctrlinfo_init(&usbctrlinfo);
+    usbctrl.devid = usbctrl_devid;
+
+    rc = libxl_device_usbctrl_getinfo(CTX, domid, &usbctrl, &usbctrlinfo);
+    if (rc) goto out;
+
+    if (usbctrlinfo.type != LIBXL_USBCTRL_TYPE_PV) {
+        LOG(ERROR, "Unsupported USB controller type");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    /* Remove usb devices first */
+    rc = libxl__device_usbdev_list_for_usbctrl(gc, domid, usbctrl_devid,
+                                               &usbdevs, &num_usbdev);
+    if (rc) goto out;
+
+    for (i = 0; i < num_usbdev; i++) {
+        if (libxl__device_usbdev_remove(gc, domid, &usbdevs[i])) {
+            LOG(ERROR, "libxl__device_usbdev_remove failed: controller %d, "
+                "port %d", usbdevs[i].ctrl, usbdevs[i].port);
+            rc = ERROR_FAIL;
+            goto out;
+        }
+    }
+
+    libxl_device_usbctrl_dispose(&usbctrl);
+    libxl_usbctrlinfo_dispose(&usbctrlinfo);
+
+    /* Remove usbctrl */
+    libxl__initiate_device_generic_remove(egc, aodev);
+    return;
+
+out:
+    libxl_device_usbctrl_dispose(&usbctrl);
+    libxl_usbctrlinfo_dispose(&usbctrlinfo);
+    aodev->rc = rc;
+    aodev->callback(egc, aodev);
+    return;
+}
+
+static const char *vusb_be_from_xs_fe(libxl__gc *gc, const char *fe_path,
+                                      uint32_t tgt_domid)
+{
+    const char *be_path;
+    int r;
+    uint32_t be_domid, fe_domid;
+
+    r = libxl__xs_read_checked(gc, XBT_NULL, GCSPRINTF("%s/backend", fe_path),
+                               &be_path);
+    if (r || !be_path) return NULL;
+
+    /* Check to see that it has the proper form, and that fe_domid ==
+     * target domid */
+    r = sscanf(be_path, "/local/domain/%d/backend/vusb/%d",
+               &be_domid, &fe_domid);
+
+    if (r != 2 || fe_domid != tgt_domid) {
+        LOG(ERROR, "Malformed backend, refusing to use");
+        return NULL;
+    }
+
+    return be_path;
+}
+
+libxl_device_usbctrl *
+libxl_device_usbctrl_list(libxl_ctx *ctx, uint32_t domid, int *num)
+{
+    GC_INIT(ctx);
+    libxl_device_usbctrl *usbctrls = NULL;
+    char *path = NULL;
+    char **entry = NULL;
+    unsigned int nentries = 0;
+
+    *num = 0;
+
+    path = GCSPRINTF("%s/device/vusb",
+                     libxl__xs_get_dompath(gc, domid));
+    entry = libxl__xs_directory(gc, XBT_NULL, path, &nentries);
+
+    if (entry && nentries) {
+        usbctrls = libxl__zalloc(NOGC, sizeof(*usbctrls) * nentries);
+        libxl_device_usbctrl *usbctrl;
+        libxl_device_usbctrl *end = usbctrls + nentries;
+        for (usbctrl = usbctrls;
+             usbctrl < end;
+             usbctrl++, entry++, (*num)++) {
+            const char *tmp, *be_path, *fe_path;
+            int ret;
+
+            libxl_device_usbctrl_init(usbctrl);
+            usbctrl->devid = atoi(*entry);
+
+#define READ_SUBPATH(path, subpath) ({                                  \
+        ret = libxl__xs_read_checked(gc, XBT_NULL,                      \
+                                     GCSPRINTF("%s/" subpath, path),    \
+                                     &tmp);                             \
+        if (ret) goto out;                                              \
+        (char *)tmp;                                                    \
+    })
+
+#define READ_SUBPATH_INT(path, subpath) ({                              \
+        ret = libxl__xs_read_checked(gc, XBT_NULL,                      \
+                                     GCSPRINTF("%s/" subpath, path),    \
+                                     &tmp);                             \
+        if (ret) goto out;                                              \
+        tmp ? atoi(tmp) : -1;                                           \
+    })
+
+            fe_path = GCSPRINTF("%s/%s", path, *entry);
+            be_path = vusb_be_from_xs_fe(gc, fe_path, domid);
+            if (!be_path) goto out;
+            usbctrl->backend_domid = READ_SUBPATH_INT(fe_path, "backend-id");
+            usbctrl->version = READ_SUBPATH_INT(be_path, "usb-ver");
+            usbctrl->ports = READ_SUBPATH_INT(be_path, "num-ports");
+            libxl_usbctrl_type_from_string(READ_SUBPATH(be_path, "type"),
+                                           &usbctrl->type);
+
+#undef READ_SUBPATH
+#undef READ_SUBPATH_INT
+       }
+    }
+
+    GC_FREE;
+    return usbctrls;
+
+out:
+    LOG(ERROR, "Unable to list USB Controllers");
+    libxl_device_usbctrl_list_free(usbctrls, *num);
+    GC_FREE;
+    *num = 0;
+    return NULL;
+}
+
+int libxl_device_usbctrl_getinfo(libxl_ctx *ctx, uint32_t domid,
+                                 libxl_device_usbctrl *usbctrl,
+                                 libxl_usbctrlinfo *usbctrlinfo)
+{
+    GC_INIT(ctx);
+    const char *dompath, *fe_path, *be_path, *tmp;
+    int rc;
+
+    usbctrlinfo->devid = usbctrl->devid;
+
+#define READ_SUBPATH(path, subpath) ({                                  \
+        rc = libxl__xs_read_checked(gc, XBT_NULL,                       \
+                                    GCSPRINTF("%s/" subpath, path),     \
+                                    &tmp);                              \
+        if (rc) goto out;                                               \
+        (char *)tmp;                                                    \
+    })
+
+#define READ_SUBPATH_INT(path, subpath) ({                              \
+        rc = libxl__xs_read_checked(gc, XBT_NULL,                       \
+                                    GCSPRINTF("%s/" subpath, path),     \
+                                    &tmp);                              \
+        if (rc) goto out;                                               \
+        tmp ? atoi(tmp) : -1;                                           \
+    })
+
+    dompath = libxl__xs_get_dompath(gc, domid);
+    fe_path = GCSPRINTF("%s/device/vusb/%d", dompath, usbctrl->devid);
+    be_path = READ_SUBPATH(fe_path, "backend");
+    usbctrlinfo->backend = libxl__strdup(NOGC, be_path);
+    usbctrlinfo->backend_id = READ_SUBPATH_INT(fe_path, "backend-id");
+    usbctrlinfo->state = READ_SUBPATH_INT(fe_path, "state");
+    usbctrlinfo->evtch = READ_SUBPATH_INT(fe_path, "event-channel");
+    usbctrlinfo->ref_urb = READ_SUBPATH_INT(fe_path, "urb-ring-ref");
+    usbctrlinfo->ref_conn = READ_SUBPATH_INT(fe_path, "urb-ring-ref");
+    tmp = READ_SUBPATH(be_path, "frontend");
+    usbctrlinfo->frontend = libxl__strdup(NOGC, tmp);
+    usbctrlinfo->frontend_id = READ_SUBPATH_INT(be_path, "frontend-id");
+    usbctrlinfo->ports = READ_SUBPATH_INT(be_path, "num-ports");
+    usbctrlinfo->version = READ_SUBPATH_INT(be_path, "usb-ver");;
+    tmp = READ_SUBPATH(be_path, "type");
+    libxl_usbctrl_type_from_string(tmp, &usbctrlinfo->type);
+
+#undef READ_SUBPATH
+#undef READ_SUBPATH_INT
+
+    rc = 0;
+
+out:
+    GC_FREE;
+    return rc;
+}
+
+int libxl_devid_to_device_usbctrl(libxl_ctx *ctx,
+                                  uint32_t domid,
+                                  int devid,
+                                  libxl_device_usbctrl *usbctrl)
+{
+    libxl_device_usbctrl *usbctrls;
+    int nb = 0;
+    int i, rc;
+
+    usbctrls = libxl_device_usbctrl_list(ctx, domid, &nb);
+    if (!usbctrls) return ERROR_FAIL;
+
+    rc = ERROR_FAIL;
+    for (i = 0; i < nb; i++) {
+        if (devid == usbctrls[i].devid) {
+            libxl_device_usbctrl_copy(ctx, usbctrl, &usbctrls[i]);
+            rc = 0;
+            break;
+        }
+    }
+
+    libxl_device_usbctrl_list_free(usbctrls, nb);
+    return rc;
+}
+
+static void *zalloc_dirent(libxl__gc *gc, const char *dirpath)
+{
+    size_t need = offsetof(struct dirent, d_name) +
+                  pathconf(dirpath, _PC_NAME_MAX) + 1;
+
+    return libxl__zalloc(gc, need);
+}
+
+static char *usbdev_busaddr_to_busid(libxl__gc *gc, int bus, int addr)
+{
+    DIR *dir;
+    char *busid = NULL;
+    struct dirent *de_buf;
+    struct dirent *de;
+
+    /* invalid hostbus or hostaddr */
+    if (bus < 1 || addr < 1)
+        return NULL;
+
+    dir = opendir(SYSFS_USB_DEV);
+    if (!dir) {
+        LOGE(ERROR, "opendir failed: '%s'", SYSFS_USB_DEV);
+        return NULL;
+    }
+
+    de_buf = zalloc_dirent(gc, SYSFS_USB_DEV);
+
+    for (;;) {
+        char *filename;
+        void *buf;
+        int busnum = -1;
+        int devnum = -1;
+
+        int r = readdir_r(dir, de_buf, &de);
+        if (r) {
+            LOGE(ERROR, "failed to readdir %s", SYSFS_USB_DEV);
+            break;
+        }
+        if (!de)
+            break;
+
+        if (!strcmp(de->d_name, ".") ||
+            !strcmp(de->d_name, ".."))
+            continue;
+
+        filename = GCSPRINTF(SYSFS_USB_DEV "/%s/devnum", de->d_name);
+        if (!libxl__read_sysfs_file_contents(gc, filename, &buf, NULL))
+            devnum = atoi(buf);
+
+        filename = GCSPRINTF(SYSFS_USB_DEV "/%s/busnum", de->d_name);
+        if (!libxl__read_sysfs_file_contents(gc, filename, &buf, NULL))
+            busnum = atoi(buf);
+
+        if (bus == busnum && addr == devnum) {
+            busid = libxl__strdup(gc, de->d_name);
+            break;
+        }
+    }
+
+    closedir(dir);
+    return busid;
+}
+
+static int usbdev_busaddr_from_busid(libxl__gc *gc, const char *busid,
+                                     uint8_t *bus, uint8_t *addr)
+{
+    char *filename;
+    void *buf;
+
+    filename = GCSPRINTF(SYSFS_USB_DEV "/%s/busnum", busid);
+    if (!libxl__read_sysfs_file_contents(gc, filename, &buf, NULL))
+        *bus = atoi(buf);
+    else
+        return ERROR_FAIL;
+
+    filename = GCSPRINTF(SYSFS_USB_DEV "/%s/devnum", busid);
+    if (!libxl__read_sysfs_file_contents(gc, filename, &buf, NULL))
+        *addr = atoi(buf);
+    else
+        return ERROR_FAIL;
+
+    return 0;
+}
+
+static int get_assigned_devices(libxl__gc *gc,
+                                libxl_device_usbdev **list, int *num)
+{
+    char **domlist;
+    unsigned int ndom = 0;
+    int i, j, k;
+    int rc;
+
+    *list = NULL;
+    *num = 0;
+
+    domlist = libxl__xs_directory(gc, XBT_NULL, "/local/domain", &ndom);
+    for (i = 0; i < ndom; i++) {
+        char *path;
+        char **usbctrls;
+        unsigned int nc = 0;
+        uint32_t domid = atoi(domlist[i]);
+
+        path = GCSPRINTF("%s/device/vusb", libxl__xs_get_dompath(gc, domid));
+        usbctrls = libxl__xs_directory(gc, XBT_NULL, path, &nc);
+
+        for (j = 0; j < nc; j++) {
+            libxl_device_usbdev *tmp = NULL;
+            int nd = 0;
+
+            rc = libxl__device_usbdev_list_for_usbctrl(gc, domid,
+                                                       atoi(usbctrls[j]),
+                                                       &tmp, &nd);
+            if (rc) goto out;
+
+            if (!nd) continue;
+
+            GCREALLOC_ARRAY(*list, *num + nd);
+            for (k = 0; k < nd; k++) {
+                libxl_device_usbdev_copy(CTX, *list + *num, tmp + k);
+                (*num)++;
+            }
+        }
+    }
+
+    return 0;
+
+out:
+    LOG(ERROR, "fail to get assigned devices");
+    return rc;
+}
+
+static bool is_usbdev_in_array(libxl_device_usbdev *usbdevs, int num,
+                               libxl_device_usbdev *usbdev)
+{
+    int i;
+
+    for (i = 0; i < num; i++) {
+        if (usbdevs[i].u.hostdev.hostbus == usbdev->u.hostdev.hostbus &&
+            usbdevs[i].u.hostdev.hostaddr == usbdev->u.hostdev.hostaddr)
+            return true;
+    }
+
+    return false;
+}
+
+/* check if USB device type is assignable */
+static bool is_usbdev_assignable(libxl__gc *gc, libxl_device_usbdev *usbdev)
+{
+    int classcode;
+    char *filename;
+    void *buf = NULL;
+    char *busid = NULL;
+
+    busid = usbdev_busaddr_to_busid(gc, usbdev->u.hostdev.hostbus,
+                                    usbdev->u.hostdev.hostaddr);
+    if (!busid) return false;
+
+    filename = GCSPRINTF(SYSFS_USB_DEV "/%s/bDeviceClass", busid);
+    if (libxl__read_sysfs_file_contents(gc, filename, &buf, NULL))
+        return false;
+
+    classcode = atoi(buf);
+    return classcode != USBHUB_CLASS_CODE;
+}
+
+/* get usb devices under certain usb controller */
+static int
+libxl__device_usbdev_list_for_usbctrl(libxl__gc *gc,
+                                      uint32_t domid,
+                                      libxl_devid usbctrl,
+                                      libxl_device_usbdev **usbdevs,
+                                      int *num)
+{
+    const char *fe_path, *be_path, *num_devs;
+    int n, i, rc;
+
+    *usbdevs = NULL;
+    *num = 0;
+
+    fe_path = GCSPRINTF("%s/device/vusb/%d",
+                        libxl__xs_get_dompath(gc, domid), usbctrl);
+
+    be_path = vusb_be_from_xs_fe(gc, fe_path, domid);
+    if (!be_path) {
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = libxl__xs_read_checked(gc, XBT_NULL,
+                                GCSPRINTF("%s/num-ports", be_path),
+                                &num_devs);
+    if (rc) goto out;
+
+    n = num_devs ? atoi(num_devs) : 0;
+
+    for (i = 0; i < n; i++) {
+        const char *busid;
+        libxl_device_usbdev *usbdev;
+
+        rc = libxl__xs_read_checked(gc, XBT_NULL,
+                                    GCSPRINTF("%s/port/%d", be_path, i + 1),
+                                    &busid);
+        if (rc) goto out;
+
+        if (busid && strcmp(busid, "")) {
+            GCREALLOC_ARRAY(*usbdevs, *num + 1);
+            usbdev = *usbdevs + *num;
+            (*num)++;
+            libxl_device_usbdev_init(usbdev);
+            usbdev->ctrl = usbctrl;
+            usbdev->port = i + 1;
+            usbdev->type = LIBXL_USBDEV_TYPE_HOSTDEV;
+            rc = usbdev_busaddr_from_busid(gc, busid,
+                                           &usbdev->u.hostdev.hostbus,
+                                           &usbdev->u.hostdev.hostaddr);
+            if (rc) goto out;
+        }
+    }
+
+    rc = 0;
+
+out:
+    return rc;
+}
+
+/* get all usb devices of the domain */
+libxl_device_usbdev *
+libxl_device_usbdev_list(libxl_ctx *ctx, uint32_t domid, int *num)
+{
+    GC_INIT(ctx);
+    libxl_device_usbdev *usbdevs = NULL;
+    const char *path;
+    char **usbctrls;
+    unsigned int nc = 0;
+    int i, j;
+
+    *num = 0;
+
+    path = GCSPRINTF("%s/device/vusb",
+                        libxl__xs_get_dompath(gc, domid));
+    usbctrls = libxl__xs_directory(gc, XBT_NULL, path, &nc);
+
+    for (i = 0; i < nc; i++) {
+        int r, nd = 0;
+        libxl_device_usbdev *tmp = NULL;
+
+        r = libxl__device_usbdev_list_for_usbctrl(gc, domid,
+                                                  atoi(usbctrls[i]),
+                                                  &tmp, &nd);
+        if (!r || !nd) continue;
+
+        usbdevs = libxl__realloc(NOGC, usbdevs,
+                                 sizeof(*usbdevs) * (*num + nd));
+        for (j = 0; j < nd; j++) {
+            libxl_device_usbdev_copy(ctx, usbdevs + *num, tmp + j);
+            (*num)++;
+        }
+    }
+
+    GC_FREE;
+    return usbdevs;
+}
+
+/* find first unused controller:port and give that to usb device */
+static int
+libxl__device_usbdev_set_default_usbctrl(libxl__gc *gc, uint32_t domid,
+                                         libxl_device_usbdev *usbdev)
+{
+    libxl_device_usbctrl *usbctrls = NULL;
+    int numctrl = 0;
+    int i, j, rc;
+
+    usbctrls = libxl_device_usbctrl_list(CTX, domid, &numctrl);
+    if (!numctrl || !usbctrls) {
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    for (i = 0; i < numctrl; i++) {
+        for (j = 0; j < usbctrls[i].ports; j++) {
+            const char *path, *tmp;
+
+            path = GCSPRINTF("%s/backend/vusb/%d/%d/port/%d",
+                             libxl__xs_get_dompath(gc, LIBXL_TOOLSTACK_DOMID),
+                             domid, usbctrls[i].devid, j + 1);
+            rc = libxl__xs_read_checked(gc, XBT_NULL, path, &tmp);
+            if (rc) goto out;
+
+            if (tmp && !strcmp(tmp, "")) {
+                usbdev->ctrl = usbctrls[i].devid;
+                usbdev->port = j + 1;
+                rc = 0;
+                goto out;
+            }
+        }
+    }
+
+    /* no available controller:port */
+    rc = ERROR_FAIL;
+
+out:
+    libxl_device_usbctrl_list_free(usbctrls, numctrl);
+    return rc;
+}
+
+/* Fill in usb information with default value.
+ *
+ * Generally, it does:
+ * 1) if "controller" is not specified:
+ *    - if "port" is not specified, try to find an available controller:port,
+ *      if found, use that; otherwise, create a new controller, use this
+ *      controller and its first port
+ *    - if "port" is specified, report error.
+ * 2) if "controller" is specified, but port is not specified:
+ *    try to find an available port under this controller, if found, use
+ *    that, otherwise, report error.
+ * 3) if both "controller" and "port" are specified:
+ *    check the controller:port is available, if not, report error.
+ */
+static int libxl__device_usbdev_setdefault(libxl__gc *gc,
+                                           uint32_t domid,
+                                           libxl_device_usbdev *usbdev,
+                                           bool update_json)
+{
+    int rc;
+
+    if (!usbdev->type)
+        usbdev->type = LIBXL_USBDEV_TYPE_HOSTDEV;
+
+    if (usbdev->ctrl == -1) {
+        if (usbdev->port) {
+            LOG(ERROR, "USB controller must be specified if you specify port");
+            return ERROR_INVAL;
+        }
+
+        rc = libxl__device_usbdev_set_default_usbctrl(gc, domid, usbdev);
+        /* If no existing controller to host this usb device, add a new one */
+        if (rc) {
+            libxl_device_usbctrl *usbctrl;
+
+            GCNEW(usbctrl);
+            libxl_device_usbctrl_init(usbctrl);
+            rc = libxl__device_usbctrl_setdefault(gc, domid, usbctrl);
+            if (rc < 0) goto out;
+
+            if (usbctrl->devid == -1) {
+                usbctrl->devid = libxl__device_nextid(gc, domid, "vusb");
+                if (usbctrl->devid < 0) {
+                    rc = ERROR_FAIL;
+                    goto out;
+                }
+            }
+
+            rc = libxl__device_usbctrl_add_xenstore(gc, domid, usbctrl,
+                                                    update_json);
+            if (rc) goto out;
+
+            usbdev->ctrl = usbctrl->devid;
+            usbdev->port = 1;
+        }
+    } else {
+        /* A controller was specified; look it up */
+        const char *fe_path, *be_path, *tmp;
+
+        fe_path = GCSPRINTF("%s/device/vusb/%d",
+                            libxl__xs_get_dompath(gc, domid),
+                            usbdev->ctrl);
+
+        be_path = vusb_be_from_xs_fe(gc, fe_path, domid);
+        if (!be_path) {
+            rc = ERROR_FAIL;
+            goto out;
+        }
+
+        if (usbdev->port) {
+            /* A specific port was requested; see if it's available */
+            rc = libxl__xs_read_checked(gc, XBT_NULL,
+                                        GCSPRINTF("%s/port/%d",
+                                                  be_path, usbdev->port),
+                                        &tmp);
+            if (rc) goto out;
+
+            if (tmp && strcmp(tmp, "")) {
+                LOG(ERROR, "The controller port isn't available");
+                rc = ERROR_FAIL;
+                goto out;
+            }
+        } else {
+            /* No port was requested. Choose free port. */
+            int i, ports;
+
+            rc = libxl__xs_read_checked(gc, XBT_NULL,
+                                        GCSPRINTF("%s/num-ports", be_path), &tmp);
+            if (rc) goto out;
+
+            ports = tmp ? atoi(tmp) : 0;
+
+            for (i = 0; i < ports; i++) {
+                rc = libxl__xs_read_checked(gc, XBT_NULL,
+                                            GCSPRINTF("%s/port/%d", be_path, i + 1),
+                                            &tmp);
+                if (rc) goto out;
+
+                if (tmp && !strcmp(tmp, "")) {
+                    usbdev->port = i + 1;
+                    break;
+                }
+            }
+
+            if (!usbdev->port) {
+                LOG(ERROR, "No available port under specified controller");
+                rc = ERROR_FAIL;
+                goto out;
+            }
+        }
+    }
+
+    rc = 0;
+
+out:
+    return rc;
+}
+
+/* Add usb information to xenstore
+ *
+ * Adding a usb device won't create new 'vusb' device, but only write
+ * the device busid to the controller:port in xenstore.
+ */
+static int libxl__device_usbdev_add_xenstore(libxl__gc *gc, uint32_t domid,
+                                             libxl_device_usbdev *usbdev,
+                                             bool update_json)
+{
+    char *be_path, *busid;
+    int rc;
+    xs_transaction_t t = XBT_NULL;
+    libxl_domain_config d_config;
+    libxl_device_usbdev usbdev_saved;
+    libxl__domain_userdata_lock *lock = NULL;
+
+    libxl_domain_config_init(&d_config);
+    libxl_device_usbdev_init(&usbdev_saved);
+    libxl_device_usbdev_copy(CTX, &usbdev_saved, usbdev);
+
+    busid = usbdev_busaddr_to_busid(gc, usbdev->u.hostdev.hostbus,
+                                    usbdev->u.hostdev.hostaddr);
+    if (!busid) {
+        LOG(DEBUG, "Fail to get busid of usb device");
+        goto out;
+    }
+
+    if (update_json) {
+        lock = libxl__lock_domain_userdata(gc, domid);
+        if (!lock) {
+            rc = ERROR_LOCK_FAIL;
+            goto out;
+        }
+
+        rc = libxl__get_domain_configuration(gc, domid, &d_config);
+        if (rc) goto out;
+
+        DEVICE_ADD(usbdev, usbdevs, domid, &usbdev_saved,
+                   COMPARE_USB, &d_config);
+    }
+
+    for (;;) {
+        rc = libxl__xs_transaction_start(gc, &t);
+        if (rc) goto out;
+
+        if (update_json) {
+            rc = libxl__set_domain_configuration(gc, domid, &d_config);
+            if (rc) goto out;
+        }
+
+        be_path = GCSPRINTF("%s/backend/vusb/%d/%d/port/%d",
+                            libxl__xs_get_dompath(gc, LIBXL_TOOLSTACK_DOMID),
+                            domid, usbdev->ctrl, usbdev->port);
+
+        LOG(DEBUG, "Adding usb device %s to xenstore: controller %d, port %d",
+            busid, usbdev->ctrl, usbdev->port);
+
+        rc = libxl__xs_write_checked(gc, t, be_path, busid);
+        if (rc) goto out;
+
+        rc = libxl__xs_transaction_commit(gc, &t);
+        if (!rc) break;
+        if (rc < 0) goto out;
+    }
+
+    rc = 0;
+
+out:
+    if (lock) libxl__unlock_domain_userdata(lock);
+    libxl_device_usbdev_dispose(&usbdev_saved);
+    libxl_domain_config_dispose(&d_config);
+    return rc;
+}
+
+static int libxl__device_usbdev_remove_xenstore(libxl__gc *gc, uint32_t domid,
+                                                libxl_device_usbdev *usbdev)
+{
+    char *be_path;
+
+    be_path = GCSPRINTF("%s/backend/vusb/%d/%d/port/%d",
+                        libxl__xs_get_dompath(gc, LIBXL_TOOLSTACK_DOMID),
+                        domid, usbdev->ctrl, usbdev->port);
+
+    LOG(DEBUG, "Removing usb device from xenstore: controller %d, port %d",
+        usbdev->ctrl, usbdev->port);
+
+    return libxl__xs_write_checked(gc, XBT_NULL, be_path, "");
+}
+
+static char *usbdev_busid_from_ctrlport(libxl__gc *gc, uint32_t domid,
+                                        libxl_device_usbdev *usbdev)
+{
+    return libxl__xs_read(gc, XBT_NULL,
+                          GCSPRINTF("%s/backend/vusb/%d/%d/port/%d",
+                              libxl__xs_get_dompath(gc, LIBXL_TOOLSTACK_DOMID),
+                          domid, usbdev->ctrl, usbdev->port));
+}
+
+/* get original driver path of usb interface, stored in @drvpath */
+static int usbintf_get_drvpath(libxl__gc *gc, const char *intf, char **drvpath)
+{
+    char *spath, *dp = NULL;
+    struct stat st;
+    int rc;
+
+    spath = GCSPRINTF(SYSFS_USB_DEV "/%s/driver", intf);
+
+    rc = lstat(spath, &st);
+    if (rc == 0) {
+        /* Find the canonical path to the driver. */
+        dp = libxl__zalloc(gc, PATH_MAX);
+        dp = realpath(spath, dp);
+        if (!dp) {
+            LOGE(ERROR, "get realpath failed: '%s'", spath);
+            return ERROR_FAIL;
+        }
+    } else if (errno != ENOENT) {
+        LOGE(ERROR, "lstat failed: '%s'", spath);
+        return ERROR_FAIL;
+    }
+
+    *drvpath = dp;
+
+    return 0;
+}
+
+static int sysfs_write_intf(libxl__gc *gc, const char *intf, const char *path)
+{
+    int fd = -1;
+    int r, rc;
+
+    fd = open(path, O_WRONLY);
+    if (fd < 0) {
+        LOGE(ERROR, "open file failed: '%s'", path);
+        return ERROR_FAIL;
+    }
+
+    r = write(fd, intf, strlen(intf));
+
+    if (r < 0) {
+        LOGE(ERROR, "write '%s' to '%s' failed", intf, path);
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    if (r != strlen(intf)) {
+        LOG(ERROR, "write '%s' to '%s' failed: incorrect write count",
+            intf, path);
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = 0;
+
+out:
+    if (fd >= 0) close(fd);
+    return rc;
+}
+
+static int unbind_usbintf(libxl__gc *gc, const char *intf)
+{
+    char *path;
+
+    path = GCSPRINTF(SYSFS_USB_DEV "/%s/driver/unbind", intf);
+    return sysfs_write_intf(gc, intf, path);
+}
+
+static int bind_usbintf(libxl__gc *gc, const char *intf, const char *drvpath)
+{
+    char *path;
+    struct stat st;
+
+    path = GCSPRINTF("%s/%s", drvpath, intf);
+    /* if already bound, return */
+    if (!lstat(path, &st))
+        return 0;
+    else if (errno != ENOENT)
+        return ERROR_FAIL;
+
+    path = GCSPRINTF("%s/bind", drvpath);
+    return sysfs_write_intf(gc, intf, path);
+}
+
+/* Is usb interface bound to usbback? */
+static int usbintf_is_assigned(libxl__gc *gc, char *intf)
+{
+    char *spath;
+    int rc;
+    struct stat st;
+
+    spath = GCSPRINTF(SYSFS_USBBACK_DRIVER "/%s", intf);
+    rc = lstat(spath, &st);
+
+    if (rc == 0)
+        return 1;
+    if (rc < 0 && errno == ENOENT)
+        return 0;
+    LOGE(ERROR, "Accessing %s", spath);
+    return -1;
+}
+
+static int usbdev_get_all_interfaces(libxl__gc *gc, const char *busid,
+                                     char ***intfs, int *num)
+{
+    DIR *dir;
+    char *buf;
+    struct dirent *de_buf;
+    struct dirent *de;
+    int rc;
+
+    *intfs = NULL;
+    *num = 0;
+
+    buf = GCSPRINTF("%s:", busid);
+
+    dir = opendir(SYSFS_USB_DEV);
+    if (!dir) {
+        LOGE(ERROR, "opendir failed: '%s'", SYSFS_USB_DEV);
+        return ERROR_FAIL;
+    }
+
+    de_buf = zalloc_dirent(gc, SYSFS_USB_DEV);
+
+    for (;;) {
+        int r = readdir_r(dir, de_buf, &de);
+
+        if (r) {
+            LOGE(ERROR, "failed to readdir %s", SYSFS_USB_DEV);
+            rc = ERROR_FAIL;
+            goto out;
+        }
+        if (!de)
+            break;
+
+        if (!strcmp(de->d_name, ".") ||
+            !strcmp(de->d_name, ".."))
+            continue;
+
+        if (!strncmp(de->d_name, buf, strlen(buf))) {
+            GCREALLOC_ARRAY(*intfs, *num + 1);
+            (*intfs)[*num] = libxl__strdup(gc, de->d_name);
+            (*num)++;
+        }
+    }
+
+    rc = 0;
+
+out:
+    closedir(dir);
+    return rc;
+}
+
+/* Encode usb interface so that it could be written to xenstore as a key.
+ *
+ * Since xenstore key cannot include '.' or ':', we'll change '.' to '_',
+ * change ':' to '@'. For example, 3-1:2.1 will be encoded to 3-1@2_1.
+ * This will be used to save original driver of USB device to xenstore.
+ */
+static char *usb_interface_xenstore_encode(libxl__gc *gc, const char *busid)
+{
+    char *str = libxl__strdup(gc, busid);
+    int i, len = strlen(str);
+
+    for (i = 0; i < len; i++) {
+        if (str[i] == '.') str[i] = '_';
+        if (str[i] == ':') str[i] = '@';
+    }
+    return str;
+}
+
+/* Unbind USB device from "usbback" driver.
+ *
+ * If there are many interfaces under USB device, check each interface,
+ * unbind from "usbback" driver and rebind to its original driver.
+ */
+static int usbback_dev_unassign(libxl__gc *gc, const char *busid)
+{
+    char **intfs = NULL;
+    char *usbdev_encode = NULL;
+    char *path = NULL;
+    int i, num = 0;
+    int rc;
+
+    rc = usbdev_get_all_interfaces(gc, busid, &intfs, &num);
+    if (rc) goto out;
+
+    usbdev_encode = usb_interface_xenstore_encode(gc, busid);
+
+    for (i = 0; i < num; i++) {
+        char *intf = intfs[i];
+        char *usbintf_encode = NULL;
+        const char *drvpath;
+
+        /* check if the USB interface is already bound to "usbback" */
+        if (usbintf_is_assigned(gc, intf) > 0) {
+            /* unbind interface from usbback driver */
+            rc = unbind_usbintf(gc, intf);
+            if (rc) goto out;
+        }
+
+        /* try to rebind USB interface to its originial driver.
+         * If rebinding failed, export warning so that user can
+         * handle it later.
+         */
+        usbintf_encode = usb_interface_xenstore_encode(gc, intf);
+        path = GCSPRINTF(USBBACK_INFO_PATH "/%s/%s/driver_path",
+                         usbdev_encode, usbintf_encode);
+        rc = libxl__xs_read_checked(gc, XBT_NULL, path, &drvpath);
+        if (rc) continue;
+
+        if (drvpath && bind_usbintf(gc, intf, drvpath))
+            LOGE(WARN, "Couldn't rebind %s to %s", intf, drvpath);
+    }
+
+    /* finally, remove xenstore driver path. */
+    path = GCSPRINTF(USBBACK_INFO_PATH "/%s", usbdev_encode);
+    rc = libxl__xs_rm_checked(gc, XBT_NULL, path);
+    if (rc)
+        LOG(WARN, "Failed to remove driver path");
+
+    /* Till here, USB device has been unbound from USBBACK and
+     * removed from xenstore, usb list couldn't show it anymore,
+     * so no matter removimg driver path successfully or not,
+     * we will report operation success.
+     */
+
+    rc = 0;
+
+out:
+    return rc;
+}
+
+/* Bind USB device to "usbback" driver.
+ *
+ * If there are many interfaces under USB device, check each interface,
+ * unbind from original driver and bind to "usbback" driver.
+ */
+static int usbback_dev_assign(libxl__gc *gc, const char *busid)
+{
+    char **intfs = NULL;
+    int num = 0, i;
+    int rc;
+    char *usbdev_encode = NULL;
+
+    rc = usbdev_get_all_interfaces(gc, busid, &intfs, &num);
+    if (rc) return rc;
+
+    usbdev_encode = usb_interface_xenstore_encode(gc, busid);
+
+    for (i = 0; i < num; i++) {
+        char *intf = intfs[i];
+        char *drvpath = NULL;
+
+        /* already assigned to usbback */
+        if (usbintf_is_assigned(gc, intf) > 0)
+            continue;
+
+        rc = usbintf_get_drvpath(gc, intf, &drvpath);
+        if (rc) goto out;
+
+        if (drvpath) {
+            /* write driver path to xenstore for later rebinding */
+            char *usbintf_encode = NULL;
+            char *path;
+
+            usbintf_encode = usb_interface_xenstore_encode(gc, intf);
+            path = GCSPRINTF(USBBACK_INFO_PATH "/%s/%s/driver_path",
+                             usbdev_encode, usbintf_encode);
+            rc = libxl__xs_write_checked(gc, XBT_NULL, path, drvpath);
+            if (rc) goto out;
+
+            /* unbind interface from original driver */
+            rc = unbind_usbintf(gc, intf);
+            if (rc) goto out;
+        }
+
+        /* bind interface to usbback */
+        rc = bind_usbintf(gc, intf, SYSFS_USBBACK_DRIVER);
+        if (rc) {
+            LOG(ERROR, "Couldn't bind %s to %s", intf, SYSFS_USBBACK_DRIVER);
+            goto out;
+        }
+    }
+
+    return 0;
+
+out:
+    /* some interfaces might be bound to usbback, unbind it and
+     * rebind it to its original driver
+     */
+    usbback_dev_unassign(gc, busid);
+    return rc;
+}
+
+static int do_usbdev_add(libxl__gc *gc, uint32_t domid,
+                         libxl_device_usbdev *usbdev,
+                         bool update_json)
+{
+    int rc;
+    char *busid;
+    libxl_device_usbctrl usbctrl;
+    libxl_usbctrlinfo usbctrlinfo;
+
+    libxl_device_usbctrl_init(&usbctrl);
+    libxl_usbctrlinfo_init(&usbctrlinfo);
+    usbctrl.devid = usbdev->ctrl;
+
+    rc = libxl_device_usbctrl_getinfo(CTX, domid, &usbctrl, &usbctrlinfo);
+    if (rc) goto out;
+
+    switch (usbctrlinfo.type) {
+    case LIBXL_USBCTRL_TYPE_PV:
+        busid = usbdev_busaddr_to_busid(gc, usbdev->u.hostdev.hostbus,
+                                        usbdev->u.hostdev.hostaddr);
+        if (!busid) {
+            rc = ERROR_FAIL;
+            goto out;
+        }
+
+        rc = libxl__device_usbdev_add_xenstore(gc, domid, usbdev, update_json);
+        if (rc) goto out;
+
+        rc = usbback_dev_assign(gc, busid);
+        if (rc) {
+            libxl__device_usbdev_remove_xenstore(gc, domid, usbdev);
+            goto out;
+        }
+        break;
+    case LIBXL_USBCTRL_TYPE_DEVICEMODEL:
+    default:
+        LOG(ERROR, "Unsupported usb controller type");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = 0;
+
+out:
+    libxl_device_usbctrl_dispose(&usbctrl);
+    libxl_usbctrlinfo_dispose(&usbctrlinfo);
+    return rc;
+}
+
+/* AO operation to add a usb device.
+ *
+ * Generally, it does:
+ * 1) check if the usb device type is assignable
+ * 2) check if the usb device is already assigned to a domain
+ * 3) add 'busid' of the usb device to xenstore contoller/port/.
+ *    (PVUSB driver watches the xenstore changes and will detect that.)
+ * 4) unbind usb device from original driver and bind to usbback.
+ *    If usb device has many interfaces, then:
+ *    - unbind each interface from its original driver and bind to usbback.
+ *    - store the original driver to xenstore for later rebinding when
+ *      detaching the device.
+ *
+ * Before calling this function, aodev should be properly filled:
+ * aodev->ao, aodev->callback, aodev->update_json, ...
+ */
+void libxl__device_usbdev_add(libxl__egc *egc, uint32_t domid,
+                              libxl_device_usbdev *usbdev,
+                              libxl__ao_device *aodev)
+{
+    STATE_AO_GC(aodev->ao);
+    int rc;
+    libxl_device_usbdev *assigned;
+    int num_assigned;
+    libxl_device_usbctrl usbctrl;
+    libxl_usbctrlinfo usbctrlinfo;
+
+    libxl_device_usbctrl_init(&usbctrl);
+    libxl_usbctrlinfo_init(&usbctrlinfo);
+
+    /* Currently only support adding USB device from Dom0 backend.
+     * So, if USB controller is specified, check its backend domain,
+     * if it's not Dom0, report error.
+     */
+    if (usbdev->ctrl != -1) {
+        usbctrl.devid = usbdev->ctrl;
+        rc = libxl_device_usbctrl_getinfo(CTX, domid, &usbctrl, &usbctrlinfo);
+        if (rc) goto out;
+
+        if (usbctrlinfo.backend_id != LIBXL_TOOLSTACK_DOMID) {
+            LOG(ERROR, "Don't support adding USB device from non-Dom0 backend");
+            rc = ERROR_INVAL;
+            goto out;
+        }
+    }
+
+    /* check usb device is assignable type */
+    if (!is_usbdev_assignable(gc, usbdev)) {
+        LOG(ERROR, "USB device is not assignable.");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    /* check usb device is already assigned */
+    rc = get_assigned_devices(gc, &assigned, &num_assigned);
+    if (rc) {
+        LOG(ERROR, "cannot determine if device is assigned,"
+                   " refusing to continue");
+        goto out;
+    }
+
+    if (is_usbdev_in_array(assigned, num_assigned, usbdev)) {
+        LOG(ERROR, "USB device already attached to a domain");
+        rc = ERROR_INVAL;
+        goto out;
+    }
+
+    /* fill default values, e.g, if usbdev->ctrl and usbdev->port
+     * not specified, choose available controller:port and fill in. */
+    rc = libxl__device_usbdev_setdefault(gc, domid, usbdev,
+                                         aodev->update_json);
+    if (rc) goto out;
+
+    /* do actual adding usb device operation */
+    rc = do_usbdev_add(gc, domid, usbdev, aodev->update_json);
+
+out:
+    libxl_device_usbctrl_dispose(&usbctrl);
+    libxl_usbctrlinfo_dispose(&usbctrlinfo);
+    aodev->rc = rc;
+    aodev->callback(egc, aodev);
+    return;
+}
+
+static int do_usbdev_remove(libxl__gc *gc, uint32_t domid,
+                            libxl_device_usbdev *usbdev)
+{
+    int rc;
+    char *busid;
+    libxl_device_usbctrl usbctrl;
+    libxl_usbctrlinfo usbctrlinfo;
+
+    libxl_device_usbctrl_init(&usbctrl);
+    libxl_usbctrlinfo_init(&usbctrlinfo);
+    usbctrl.devid = usbdev->ctrl;
+
+    rc = libxl_device_usbctrl_getinfo(CTX, domid, &usbctrl, &usbctrlinfo);
+    if (rc) goto out;
+
+    switch (usbctrlinfo.type) {
+    case LIBXL_USBCTRL_TYPE_PV:
+        busid = usbdev_busid_from_ctrlport(gc, domid, usbdev);
+        if (!busid) {
+            rc = ERROR_FAIL;
+            goto out;
+        }
+
+        rc = libxl__device_usbdev_remove_xenstore(gc, domid, usbdev);
+        if (rc) goto out;
+
+        rc = usbback_dev_unassign(gc, busid);
+        if (rc) {
+            /* Till here, usb device information is already removed
+             * from xenstore, usb list couldn't list it any more.
+             * If unassign usb device from usbback failed, export
+             * warning only so that user could handle driver status
+             * later.
+             */
+            LOG(WARN, "unbind usb device from usbback and rebind to its "
+                      "original driver failed");
+        }
+        break;
+    case LIBXL_USBCTRL_TYPE_DEVICEMODEL:
+    default:
+        LOG(ERROR, "Unsupported usb controller type");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = 0;
+
+out:
+    libxl_device_usbctrl_dispose(&usbctrl);
+    libxl_usbctrlinfo_dispose(&usbctrlinfo);
+    return rc;
+}
+
+/* Operation to remove usb device.
+ *
+ * Generally, it does:
+ * 1) check if the usb device is assigned to the domain
+ * 2) remove the usb device from xenstore controller/port.
+ * 3) unbind usb device from usbback and rebind to its original driver.
+ *    If usb device has many interfaces, do it to each interface.
+ */
+static int libxl__device_usbdev_remove(libxl__gc *gc, uint32_t domid,
+                                       libxl_device_usbdev *usbdev)
+{
+    libxl_usbctrlinfo usbctrlinfo;
+    libxl_device_usbctrl usbctrl;
+    int rc;
+
+    if (usbdev->ctrl < 0 || usbdev->port < 1) {
+        LOG(ERROR, "Invalid USB device");
+        return ERROR_FAIL;
+    }
+
+    libxl_device_usbctrl_init(&usbctrl);
+    libxl_usbctrlinfo_init(&usbctrlinfo);
+    usbctrl.devid = usbdev->ctrl;
+
+    rc = libxl_device_usbctrl_getinfo(CTX, domid, &usbctrl, &usbctrlinfo);
+    if (rc) goto out;
+
+    if (usbctrlinfo.backend_id != LIBXL_TOOLSTACK_DOMID) {
+        LOG(ERROR, "Don't support removing USB device from non-Dom0 backend");
+        rc = ERROR_INVAL;
+        goto out;
+    }
+
+    /* do actual removing usb device operation */
+    rc = do_usbdev_remove(gc, domid, usbdev);
+
+out:
+    libxl_device_usbctrl_dispose(&usbctrl);
+    libxl_usbctrlinfo_dispose(&usbctrlinfo);
+    return rc;
+}
+
+int libxl_device_usbdev_remove(libxl_ctx *ctx, uint32_t domid,
+                               libxl_device_usbdev *usbdev,
+                               const libxl_asyncop_how *ao_how)
+
+{
+    AO_CREATE(ctx, domid, ao_how);
+    int rc;
+
+    rc = libxl__device_usbdev_remove(gc, domid, usbdev);
+
+    libxl__ao_complete(egc, ao, rc);
+    return AO_INPROGRESS;
+}
+
+int libxl_ctrlport_to_device_usbdev(libxl_ctx *ctx,
+                                    uint32_t domid,
+                                    int ctrl,
+                                    int port,
+                                    libxl_device_usbdev *usbdev)
+{
+    GC_INIT(ctx);
+    const char *dompath, *fe_path, *be_path, *busid;
+    int rc;
+
+    dompath = libxl__xs_get_dompath(gc, domid);
+
+    fe_path = GCSPRINTF("%s/device/vusb/%d", dompath, ctrl);
+
+    be_path = vusb_be_from_xs_fe(gc, fe_path, domid);
+    if (!be_path) {
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = libxl__xs_read_checked(gc, XBT_NULL,
+                           GCSPRINTF("%s/port/%d", be_path, port),
+                           &busid);
+    if (rc) goto out;
+
+    if (!busid || !strcmp(busid, "")) {
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    usbdev->ctrl = ctrl;
+    usbdev->port = port;
+    usbdev->type = LIBXL_USBDEV_TYPE_HOSTDEV;
+    rc = usbdev_busaddr_from_busid(gc, busid,
+                                   &usbdev->u.hostdev.hostbus,
+                                   &usbdev->u.hostdev.hostaddr);
+
+out:
+    GC_FREE;
+    return rc;
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 9658356..befee94 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -606,6 +606,35 @@ libxl_device_rdm = Struct("device_rdm", [
     ("policy", libxl_rdm_reserve_policy),
     ])
 
+libxl_usbctrl_type = Enumeration("usbctrl_type", [
+    (0, "AUTO"),
+    (1, "PV"),
+    (2, "DEVICEMODEL"),
+    ])
+
+libxl_usbdev_type = Enumeration("usbdev_type", [
+    (1, "hostdev"),
+    ])
+
+libxl_device_usbctrl = Struct("device_usbctrl", [
+    ("type", libxl_usbctrl_type),
+    ("devid", libxl_devid),
+    ("version", integer),
+    ("ports", integer),
+    ("backend_domid", libxl_domid),
+    ("backend_domname", string),
+   ])
+
+libxl_device_usbdev = Struct("device_usbdev", [
+    ("ctrl", libxl_devid),
+    ("port", integer),
+    ("u", KeyedUnion(None, libxl_usbdev_type, "type",
+           [("hostdev", Struct(None, [
+                 ("hostbus",   uint8),
+                 ("hostaddr",  uint8)])),
+           ])),
+    ])
+
 libxl_device_dtdev = Struct("device_dtdev", [
     ("path", string),
     ])
@@ -644,6 +673,8 @@ libxl_domain_config = Struct("domain_config", [
     # a channel manifests as a console with a name,
     # see docs/misc/channels.txt
     ("channels", Array(libxl_device_channel, "num_channels")),
+    ("usbctrls", Array(libxl_device_usbctrl, "num_usbctrls")),
+    ("usbdevs", Array(libxl_device_usbdev, "num_usbdevs")),
 
     ("on_poweroff", libxl_action_on_shutdown),
     ("on_reboot", libxl_action_on_shutdown),
@@ -687,6 +718,21 @@ libxl_vtpminfo = Struct("vtpminfo", [
     ("uuid", libxl_uuid),
     ], dir=DIR_OUT)
 
+libxl_usbctrlinfo = Struct("usbctrlinfo", [
+    ("type", libxl_usbctrl_type),
+    ("devid", libxl_devid),
+    ("version", integer),
+    ("ports", integer),
+    ("backend", string),
+    ("backend_id", uint32),
+    ("frontend", string),
+    ("frontend_id", uint32),
+    ("state", integer),
+    ("evtch", integer),
+    ("ref_urb", integer),
+    ("ref_conn", integer),
+    ], dir=DIR_OUT)
+
 libxl_vcpuinfo = Struct("vcpuinfo", [
     ("vcpuid", uint32),
     ("cpu", uint32),
diff --git a/tools/libxl/libxl_types_internal.idl b/tools/libxl/libxl_types_internal.idl
index 5e55685..696f5f8 100644
--- a/tools/libxl/libxl_types_internal.idl
+++ b/tools/libxl/libxl_types_internal.idl
@@ -22,6 +22,7 @@ libxl__device_kind = Enumeration("device_kind", [
     (6, "VKBD"),
     (7, "CONSOLE"),
     (8, "VTPM"),
+    (9, "VUSB"),
     ])
 
 libxl__console_backend = Enumeration("console_backend", [
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index e64f301..94dac4e 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -1311,6 +1311,24 @@ int libxl__random_bytes(libxl__gc *gc, uint8_t *buf, size_t len)
     return ret;
 }
 
+void libxl_device_usbctrl_list_free(libxl_device_usbctrl *list, int nr)
+{
+   int i;
+
+   for (i = 0; i < nr; i++)
+       libxl_device_usbctrl_dispose(&list[i]);
+   free(list);
+}
+
+void libxl_device_usbdev_list_free(libxl_device_usbdev *list, int nr)
+{
+   int i;
+
+   for (i = 0; i < nr; i++)
+       libxl_device_usbdev_dispose(&list[i]);
+   free(list);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxl/libxl_utils.h b/tools/libxl/libxl_utils.h
index 339ebdf..4495417 100644
--- a/tools/libxl/libxl_utils.h
+++ b/tools/libxl/libxl_utils.h
@@ -76,6 +76,11 @@ int libxl_uuid_to_device_vtpm(libxl_ctx *ctx, uint32_t domid,
                                libxl_uuid *uuid, libxl_device_vtpm *vtpm);
 int libxl_devid_to_device_vtpm(libxl_ctx *ctx, uint32_t domid,
                                int devid, libxl_device_vtpm *vtpm);
+int libxl_devid_to_device_usbctrl(libxl_ctx *ctx, uint32_t domid,
+                                  int devid, libxl_device_usbctrl *usbctrl);
+int libxl_ctrlport_to_device_usbdev(libxl_ctx *ctx, uint32_t domid,
+                                    int ctrl, int port,
+                                    libxl_device_usbdev *usbdev);
 
 int libxl_bitmap_alloc(libxl_ctx *ctx, libxl_bitmap *bitmap, int n_bits);
     /* Allocated bimap is from malloc, libxl_bitmap_dispose() to be
-- 
2.1.4