xl PVUSB pass-through

xl PVUSB pass-through

[Stefan]

[-- Attachment #1.1: Type: text/plain, Size: 1555 bytes --]

Good Morning:

Our Xen clusters depend heavily on that feature. The lack of it prevents 
us from transitioning to the new toolstack. Currently we use PVUSB to 
attach a USB Smartcard reader through our dom0 (SLES 11 SP1) running on 
an HP Blade Server with the Token mounted on an internal USB Port to our 
domU CA server (SLES 11)

I'd like to know how likely it is to make it to the next release. Does 
anyone have a solution that doesn't require PVUSB? I'd like to give a 
hand if that can help making PVUSB part of Xen 4.3.

Thanks

Stefan

On 03/06/2013 05:22 AM, George Dunlap wrote:
> Stefan,
>
> Thanks for your e-mail.  Would you mind replying to my e-mail on the 
> list with this question?  One of the reasons PVUSB has been a low 
> priority is that the core Xen developers haven't seen a demand for it; 
> asking this question there can raise awareness, and may prompt other 
> people in the same situation to speak up. Additionally, there are some 
> technical details which I believe are true but I'm not 100% on -- 
> having this discussion in public ensures that you get the best 
> technical answer possible. :-)
>
>  -George
>
> Hello George,
>
> Are you in charge of implementing that feature in Xen 4.3? I found your
> email address indirectly on this page :
> http://wiki.xen.org/wiki/Xen_Roadmap/4.3
>
> Our Xen clusters depend heavily on that feature. I'd like to know how
> likely it is to make it to the next release... I'd like to give a hand
> if that can help making PVUSB part of Xen 4.3.
>
> Thanks
>
> Stefan Stanisic
>
>


[-- Attachment #1.2: Type: text/html, Size: 2365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xl PVUSB pass-through

[Pasi Kärkkäinen]

On Wed, Mar 06, 2013 at 10:59:26AM -0500, Stefan wrote:
>    Good Morning:
> 
>    Our Xen clusters depend heavily on that feature. The lack of it prevents
>    us from transitioning to the new toolstack. Currently we use PVUSB to
>    attach a USB Smartcard reader through our dom0 (SLES 11 SP1) running on an
>    HP Blade Server with the Token mounted on an internal USB Port to our domU
>    CA server (SLES 11)
> 
>    I'd like to know how likely it is to make it to the next release. Does
>    anyone have a solution that doesn't require PVUSB? I'd like to give a hand
>    if that can help making PVUSB part of Xen 4.3.
>

Help with PVUSB support would be very much welcome!

These are the things missing for PVUSB support:

- PVUSB frontend/backend driver cleanup and upstreaming to Linux.
- xl/libxl support for PVUSB.

The drivers probably require more work, and the xl/libxl support is the easier one.
You can find the the pvusb drivers ported to pvops 3.x kernels from Konrad's git repo.

-- Pasi



>    Thanks
> 
>    Stefan
> 
>    On 03/06/2013 05:22 AM, George Dunlap wrote:
> 
>      Stefan,
> 
>      Thanks for your e-mail.  Would you mind replying to my e-mail on the
>      list with this question?  One of the reasons PVUSB has been a low
>      priority is that the core Xen developers haven't seen a demand for it;
>      asking this question there can raise awareness, and may prompt other
>      people in the same situation to speak up.  Additionally, there are some
>      technical details which I believe are true but I'm not 100% on -- having
>      this discussion in public ensures that you get the best technical answer
>      possible. :-)
> 
>       -George
> 
>  Hello George,
> 
>  Are you in charge of implementing that feature in Xen 4.3? I found your
>  email address indirectly on this page :
>  [1]http://wiki.xen.org/wiki/Xen_Roadmap/4.3
> 
>  Our Xen clusters depend heavily on that feature. I'd like to know how
>  likely it is to make it to the next release... I'd like to give a hand
>  if that can help making PVUSB part of Xen 4.3.
> 
>  Thanks
> 
>  Stefan Stanisic
> 
> References
> 
>    Visible links
>    1. http://wiki.xen.org/wiki/Xen_Roadmap/4.3

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: xl PVUSB pass-through

[George Dunlap]

On Wed, Mar 6, 2013 at 3:59 PM, Stefan <sstanisi@cbnco.com> wrote:
> Good Morning:
>
> Our Xen clusters depend heavily on that feature. The lack of it prevents us
> from transitioning to the new toolstack. Currently we use PVUSB to attach a
> USB Smartcard reader through our dom0 (SLES 11 SP1) running on an HP Blade
> Server with the Token mounted on an internal USB Port to our domU CA server
> (SLES 11)
>
> I'd like to know how likely it is to make it to the next release. Does
> anyone have a solution that doesn't require PVUSB? I'd like to give a hand
> if that can help making PVUSB part of Xen 4.3.

For PVUSB to work for any particular VM, you need two things:
* PVUSB kernel support in the guest VM and dom0 (or driver domain)
* The toolstack has to know how to connect the two.

"Classic xen" kernels like those supported by SuSE have PVUSB support,
and xend has support for setting up the connection.

So regarding PVUSB there are two tasks:

1. Teach xl/libxl how to set up the connection
2. Port PVUSB to pvops kernels.

It sounds like the main thing you need is to have the xl/libxl
support.  This should just be a matter of looking at what xend does in
setting up the PVUSB connection, and duplicating that in xl.  If you'd
be willing to give that a try, that would be great -- we'd love to
help out.

If you've never submitted patches before,
http://wiki.xen.org/wiki/Submitting_Xen_Patches has some helpful
advice.

If for some reason PVUSB doesn't make it into 4.3, another option you
can explore is running your VMs in PVHVM mode; xl will definitely
support USB pass-through for HVM domains in 4.3.

 -George

Re: xl PVUSB pass-through

[Pasi Kärkkäinen]

On Thu, Mar 07, 2013 at 10:07:32AM +0000, George Dunlap wrote:
> On Wed, Mar 6, 2013 at 3:59 PM, Stefan <sstanisi@cbnco.com> wrote:
> > Good Morning:
> >
> > Our Xen clusters depend heavily on that feature. The lack of it prevents us
> > from transitioning to the new toolstack. Currently we use PVUSB to attach a
> > USB Smartcard reader through our dom0 (SLES 11 SP1) running on an HP Blade
> > Server with the Token mounted on an internal USB Port to our domU CA server
> > (SLES 11)
> >
> > I'd like to know how likely it is to make it to the next release. Does
> > anyone have a solution that doesn't require PVUSB? I'd like to give a hand
> > if that can help making PVUSB part of Xen 4.3.
> 
> For PVUSB to work for any particular VM, you need two things:
> * PVUSB kernel support in the guest VM and dom0 (or driver domain)
> * The toolstack has to know how to connect the two.
> 
> "Classic xen" kernels like those supported by SuSE have PVUSB support,
> and xend has support for setting up the connection.
> 
> So regarding PVUSB there are two tasks:
> 
> 1. Teach xl/libxl how to set up the connection
> 2. Port PVUSB to pvops kernels.
> 

PVUSB drivers have already been ported to pvops kernels,
and the PVUSB drivers can be found from Konrad's git repo.

What is missing is upstreaming the drivers.


> It sounds like the main thing you need is to have the xl/libxl
> support.  This should just be a matter of looking at what xend does in
> setting up the PVUSB connection, and duplicating that in xl.  If you'd
> be willing to give that a try, that would be great -- we'd love to
> help out.
> 
> If you've never submitted patches before,
> http://wiki.xen.org/wiki/Submitting_Xen_Patches has some helpful
> advice.
> 
> If for some reason PVUSB doesn't make it into 4.3, another option you
> can explore is running your VMs in PVHVM mode; xl will definitely
> support USB pass-through for HVM domains in 4.3.
> 


-- Pasi

Re: xl PVUSB pass-through

[Marek Marczykowski]

[-- Attachment #1.1: Type: text/plain, Size: 5430 bytes --]

On 07.03.2013 15:11, Pasi Kärkkäinen wrote:
> On Thu, Mar 07, 2013 at 10:07:32AM +0000, George Dunlap wrote:
>> On Wed, Mar 6, 2013 at 3:59 PM, Stefan <sstanisi@cbnco.com> wrote:
>>> Good Morning:
>>>
>>> Our Xen clusters depend heavily on that feature. The lack of it prevents us
>>> from transitioning to the new toolstack. Currently we use PVUSB to attach a
>>> USB Smartcard reader through our dom0 (SLES 11 SP1) running on an HP Blade
>>> Server with the Token mounted on an internal USB Port to our domU CA server
>>> (SLES 11)
>>>
>>> I'd like to know how likely it is to make it to the next release. Does
>>> anyone have a solution that doesn't require PVUSB? I'd like to give a hand
>>> if that can help making PVUSB part of Xen 4.3.
>>
>> For PVUSB to work for any particular VM, you need two things:
>> * PVUSB kernel support in the guest VM and dom0 (or driver domain)
>> * The toolstack has to know how to connect the two.
>>
>> "Classic xen" kernels like those supported by SuSE have PVUSB support,
>> and xend has support for setting up the connection.
>>
>> So regarding PVUSB there are two tasks:
>>
>> 1. Teach xl/libxl how to set up the connection
>> 2. Port PVUSB to pvops kernels.
>>
> 
> PVUSB drivers have already been ported to pvops kernels,
> and the PVUSB drivers can be found from Konrad's git repo.
> 
> What is missing is upstreaming the drivers.

Actually the drivers aren't working well :(
We've tried to use them in Qubes OS, but it is definitely not production ready
yet.

Details in this thread (mostly on Qubes-specific tools, but also on kernel
pvusb drivers):
https://groups.google.com/group/qubes-devel/browse_thread/thread/e002ae940061d897/b4d63fbbb1b29b48

Some quotes (from Alexandre Bezroutchko who was testing/debugging the drivers):
"There might be more than one problem actually. The easiest to reproduce
is kernel memory management issue which reliably causes oopses in
kfree() USB Ethernet adapter is attached. This happens when the backend
tries to free memory block used for USB event handling. I started
debugging it, activated memory-related debugging options in the kernel
and it started crashing much earlier, during bootstrap."

"Another problem I saw is bus resets. When I try to pass-through USB
stick, it gets attached but it takes a minute or so before it starts
working in domU. During this time there are some bus timeout messages or
something in dmesg. This could have something to do with missed events
in frontend-backend communication. I didn't investigate it though, guess
memory management issue needs to be sorted out first."


Trying USBIP (as fallback for PVUSB) points out same problem as in PVUSB:
"The actual problem is USBIP module frees URB transfer buffers
and USB core does the same afterwards. There is a flag controlling this
behaviour (introduced in this patch [1] in kernel). Apparently this flag is
controlled remotely by the frontend, it is mentioned in usbip_protocol.txt
part of the protocol. This is odd, why it can possibly be up to the
frontend to decide whether to free the transfer buffer or not. This
transfer buffer is allocated at stub_rx.c:485, right after URB holding it
is allocated with urb_alloc_urb() at lines 472/475. Eventually the transfer
buffer is freed in stub_main.c:239 and stub_tx.c:31, right before URB is
freed with usb_free_urb(). The latter internally triggers urb_destroy()
which, if URB_FREE_BUFFER flag is set in URB, calls kfree() and actually
causes all the troubles. I think proper solution would be to mask out
URB_FREE_BUFFER from the received transfer flags. There is already some
filtering for the flags stub_rx.c:437 but URB_FREE_BUFFER is explicitly
permitted, which is odd.

By the way, it is possible that pvusb was plagued by the same issue -- at
the first glance I don't find any attempt to filter value of
transfer_flags, normally should happen somewhere around usbback.c:526, but
it does not.

If I'm not mistaken the above problem only affects the backend, so frontend
lockups something else probably. Maybe it is a side effect of my patch to
the lockup problem :). To troubleshoot stalls I figure I can try to run
debugger against the stalled kernel over a virtual serial link and dump
stack trace on the stalled CPU, this might clairfy where the problem is."

Some fix (workaround?) is here:
https://github.com/grwl/qubes-kernel/commit/4f9a1404eb00b679b507a906a8ea26dea0a4874b
But this is only one issue of many.

> 
>> It sounds like the main thing you need is to have the xl/libxl
>> support.  This should just be a matter of looking at what xend does in
>> setting up the PVUSB connection, and duplicating that in xl.  If you'd
>> be willing to give that a try, that would be great -- we'd love to
>> help out.
>>
>> If you've never submitted patches before,
>> http://wiki.xen.org/wiki/Submitting_Xen_Patches has some helpful
>> advice.
>>
>> If for some reason PVUSB doesn't make it into 4.3, another option you
>> can explore is running your VMs in PVHVM mode; xl will definitely
>> support USB pass-through for HVM domains in 4.3.
>>
> 
> 
> -- Pasi
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 


-- 
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 553 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xl PVUSB pass-through

[jacek burghardt]

[-- Attachment #1.1: Type: text/plain, Size: 279 bytes --]

Qemu supports smart cards and xen 4.3 can be compiled with upstream
qemu-xen and it allows smart card support
http://wiki.qemu.org/Features/Smartcard
Maybe another option is spice protocol as it supports usb redirection and
smart cards
http://docs.cslabs.clarkson.edu/wiki/SPICE

[-- Attachment #1.2: Type: text/html, Size: 409 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xl PVUSB pass-through

[George Dunlap]

On 11/03/13 04:59, jacek burghardt wrote:
> Qemu supports smart cards and xen 4.3 can be compiled with upstream 
> qemu-xen and it allows smart card support
> http://wiki.qemu.org/Features/Smartcard
> Maybe another option is spice protocol as it supports usb redirection 
> and smart cards
> http://docs.cslabs.clarkson.edu/wiki/SPICE

If he's using an HVM domain with qemu, he will already be able to 
pass-through USB devices.  But we will eventually want PVUSB for those 
who want to use PV without qemu.

  -George

Re: xl PVUSB pass-through

[jacek burghardt]

[-- Attachment #1.1: Type: text/plain, Size: 164 bytes --]

Well working pvusb would be nice. I believe for now that would be better
work around. I did not know was possible to run pv without qemu as it
emulates pc devices.

[-- Attachment #1.2: Type: text/html, Size: 175 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xl PVUSB pass-through

[Konrad Rzeszutek Wilk]

On Sun, Mar 10, 2013 at 11:46:50PM +0100, Marek Marczykowski wrote:
> On 07.03.2013 15:11, Pasi Kärkkäinen wrote:
> > On Thu, Mar 07, 2013 at 10:07:32AM +0000, George Dunlap wrote:
> >> On Wed, Mar 6, 2013 at 3:59 PM, Stefan <sstanisi@cbnco.com> wrote:
> >>> Good Morning:
> >>>
> >>> Our Xen clusters depend heavily on that feature. The lack of it prevents us
> >>> from transitioning to the new toolstack. Currently we use PVUSB to attach a
> >>> USB Smartcard reader through our dom0 (SLES 11 SP1) running on an HP Blade
> >>> Server with the Token mounted on an internal USB Port to our domU CA server
> >>> (SLES 11)
> >>>
> >>> I'd like to know how likely it is to make it to the next release. Does
> >>> anyone have a solution that doesn't require PVUSB? I'd like to give a hand
> >>> if that can help making PVUSB part of Xen 4.3.
> >>
> >> For PVUSB to work for any particular VM, you need two things:
> >> * PVUSB kernel support in the guest VM and dom0 (or driver domain)
> >> * The toolstack has to know how to connect the two.
> >>
> >> "Classic xen" kernels like those supported by SuSE have PVUSB support,
> >> and xend has support for setting up the connection.
> >>
> >> So regarding PVUSB there are two tasks:
> >>
> >> 1. Teach xl/libxl how to set up the connection
> >> 2. Port PVUSB to pvops kernels.
> >>
> > 
> > PVUSB drivers have already been ported to pvops kernels,
> > and the PVUSB drivers can be found from Konrad's git repo.
> > 
> > What is missing is upstreaming the drivers.
> 
> Actually the drivers aren't working well :(
> We've tried to use them in Qubes OS, but it is definitely not production ready
> yet.
> 
> Details in this thread (mostly on Qubes-specific tools, but also on kernel
> pvusb drivers):
> https://groups.google.com/group/qubes-devel/browse_thread/thread/e002ae940061d897/b4d63fbbb1b29b48
> 
> Some quotes (from Alexandre Bezroutchko who was testing/debugging the drivers):
> "There might be more than one problem actually. The easiest to reproduce
> is kernel memory management issue which reliably causes oopses in
> kfree() USB Ethernet adapter is attached. This happens when the backend
> tries to free memory block used for USB event handling. I started
> debugging it, activated memory-related debugging options in the kernel
> and it started crashing much earlier, during bootstrap."

I believe the fix for that was posted by Boris:

https://lkml.org/lkml/2013/2/26/420
> 
> "Another problem I saw is bus resets. When I try to pass-through USB
> stick, it gets attached but it takes a minute or so before it starts
> working in domU. During this time there are some bus timeout messages or
> something in dmesg. This could have something to do with missed events
> in frontend-backend communication. I didn't investigate it though, guess
> memory management issue needs to be sorted out first."
> 
> 
> Trying USBIP (as fallback for PVUSB) points out same problem as in PVUSB:
> "The actual problem is USBIP module frees URB transfer buffers
> and USB core does the same afterwards. There is a flag controlling this
> behaviour (introduced in this patch [1] in kernel). Apparently this flag is
> controlled remotely by the frontend, it is mentioned in usbip_protocol.txt
> part of the protocol. This is odd, why it can possibly be up to the
> frontend to decide whether to free the transfer buffer or not. This
> transfer buffer is allocated at stub_rx.c:485, right after URB holding it
> is allocated with urb_alloc_urb() at lines 472/475. Eventually the transfer
> buffer is freed in stub_main.c:239 and stub_tx.c:31, right before URB is
> freed with usb_free_urb(). The latter internally triggers urb_destroy()
> which, if URB_FREE_BUFFER flag is set in URB, calls kfree() and actually
> causes all the troubles. I think proper solution would be to mask out
> URB_FREE_BUFFER from the received transfer flags. There is already some
> filtering for the flags stub_rx.c:437 but URB_FREE_BUFFER is explicitly
> permitted, which is odd.
> 
> By the way, it is possible that pvusb was plagued by the same issue -- at
> the first glance I don't find any attempt to filter value of
> transfer_flags, normally should happen somewhere around usbback.c:526, but
> it does not.
> 
> If I'm not mistaken the above problem only affects the backend, so frontend
> lockups something else probably. Maybe it is a side effect of my patch to
> the lockup problem :). To troubleshoot stalls I figure I can try to run
> debugger against the stalled kernel over a virtual serial link and dump
> stack trace on the stalled CPU, this might clairfy where the problem is."
> 
> Some fix (workaround?) is here:
> https://github.com/grwl/qubes-kernel/commit/4f9a1404eb00b679b507a906a8ea26dea0a4874b
> But this is only one issue of many.
> 
> > 
> >> It sounds like the main thing you need is to have the xl/libxl
> >> support.  This should just be a matter of looking at what xend does in
> >> setting up the PVUSB connection, and duplicating that in xl.  If you'd
> >> be willing to give that a try, that would be great -- we'd love to
> >> help out.
> >>
> >> If you've never submitted patches before,
> >> http://wiki.xen.org/wiki/Submitting_Xen_Patches has some helpful
> >> advice.
> >>
> >> If for some reason PVUSB doesn't make it into 4.3, another option you
> >> can explore is running your VMs in PVHVM mode; xl will definitely
> >> support USB pass-through for HVM domains in 4.3.
> >>
> > 
> > 
> > -- Pasi
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> > 
> 
> 
> -- 
> Best Regards / Pozdrawiam,
> Marek Marczykowski
> Invisible Things Lab
> 



> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel