[Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

In QEMU there are a number of features which involve communication with an
external system over an I/O channel of some form. The features include
migration, NBD, VNC and character devices. The I/O channel in question might
might be a FIFO pipe, a PTY, a TCP socket, a UNIX domain socket, RMDA channel
or something else, while the external system can be another QEMU, libvirt, an
NBD server, or something else.

Currently the only place where there is any meaningful level of security is
the VNC server, which supports the VeNCrypt extension providing TLS sessions
for data encryption and x509 cert validation for authentication, and the SASL
extension which also provides either encryption of authentication or both.
The migration data channel is more or less completely unprotected unless it
is tunnelled via libvirt or equivalent external secure channel. The same is
true for NBD, though there was a recent discussion about defining an extension
to use TLS. Likewise serial ports, parallel ports, virtio consoles all use the
chardev backends which offer no security features.

There are some other network related pieces in QEMU, namely the built-in iSCSI,
RBD, NFS clients, and SPICE server. Since those are all implemented via 3rd
party libraries rather than natively by QEMU I'm going ignore them for the sake
of this discussion and focus on network interaction directly implemented in
QEMU.

We have a broad goal in OpenStack that every network channel in use must have
encryption and authentication capabilities. Currently all the communication
channels between the end user and the cloud infrastructure edge servers are
secured, but internally a number of the cloud infrastructure components are
unsecured. For example, we recommend to tunnel migration via libvirt, though
that excludes use of the NBD for block migration since libvirt can't currently
tunnel that. Internally we will shortly use  VeNCrypt for protecting VNC
between the compute hosts and the openstack console proxy edge servers. We
are lacking the abilty to secure serial ports between the compute nods and
console proxy.

Essentially the project considers that it is no longer sufficient to consider
the private management LAN (on which the cloud infrastructure is deployed) to
be fully trusted; it must be considered hostile.

So back to QEMU/KVM, we want to have

 - Native support for encryption & authentication with migration, since
   tunnelling via libvirt has a non-negligble performance overhead adding
   both latency and CPU load

 - Native support for encryption & authentication with NBD server to enable
   security of the block migration service

 - Native support for encryption & authentication with chardev TCP backends
   to enable security of the serial port consoles.

As a starting point, the desire is to have TLS for session encryption and
x509 certificate verification for authentication, but at a later date we'd
also like to add the ability to use SASL for either aspect too. Thinking about
QEMU users in general, it would probably be useful if any impl could allow for
future enhancements such as SSH tunnelling too.

I have some development cycles available to work on addressing these gaps in
QEMU, along with relevant experiance since I did the previous VNC server work
for adding TLS and SASL in QEMU, along with similar in libvirt and GTK-VNC.

Having looked at the QEMU code for VNC, migration and chardevs I think the
main stumbling block is currently that QEMU does not have any kind of standard
approach for dealing with I/O channels internally. Migration has the QEMUFile
abstraction, but it is currently fairly coupled to the migration code itself
and limited in scope, because it doesn't actually deal with socket listen
or connect tasks. That's still part of the migration code itself, QEMUFile
only deals with I/O transfer, so it is a pretty incomplete abstraction layer.
The chardev code has a backend abstraction, but that is really horribly
entwined with the chardev code so not reusable in any way without a rewrite
from scratch IMHO. The VNC server has alot of useful code for dealing with
TLS & SASL, but it is somewhat entwined with the VNC server. The VNC server
doesn't use any I/O abstraction just preferring raw FDs / sockets. I've not
looked at the NBD code in detail, but I'm presuming it is using raw FDs /
sockets.

Since this TLS/SASL code is non-trivial (and obviously security critical), I
really don't want to end up with 4 separate places to implement it in QEMU.
IMHO the only practical / sensible approach is to define some kind of standard
I/O channel API internally to QEMU which migration, NBD, chardev and VNC all
use. That gives us a single place to integrate all the security mechanisms
we need to support.  In libvirt we did something like this a little while
ago by defining a standard internal sockets API[1], with plugins for things
like SASL[2], TLS[4] and SSH[5] and it has been very successful in simplifying
the code by centralizing the hairy logic, though I wouldn't aim for exactly
the same design if doing it again - a more layered approach like QEMU blockdev
drivers is probably better in retrospect.

So my idea would be that we define a QEMUChannel object and set of APIs to
standardize all interaction with sockets, pipes, RDMA, whatever $channel,
and then convert the QEMU features I've mentioned over to use that. I think
that would be simpler than trying to untangle QEMUFile code from migration
and then extend its features.

A rough plan of attack would be to split the work in a number of distinct
pieces

 - Define a basic QEMUChannel object & APIs.

 - Convert chardev backend to the QEMUChannel API

 - Convert migration to the QEMUChannel API

 - Convert NBD to the QEMUChannel API

 - Introduce support for TLS to the QEMUChannel object

 - Introduce support for SASL to the QEMUChannel object

 - Convert VNC server to the QEMUChannel API

 - Integrate TLS in migration  (eg flags turn it on/off via CLI/monitor)

 - Integrate TLS in NBD (eg flags to turn it on/off via CLI/monitor)

 - Integrate TLS in chardev backend  (eg flags to turn it on/off via CLI/monitor)

Aside from the first task, I'm not sure that's the best/right order to do
the work, it is merely the rough set of pieces I see as required. I've also
no idea how long this would take. It is clearly a pretty large task given
the different areas of code it touchs.

In terms of openstack priorities though, migration is the most important,
which would be tackling migration + NBD, with chardevs being a secondary
importance. Converting VNC is obviously lowest priority since that already
has security - it is just a sanity thing to avoid having two inmpls of TLS
in QEMU long term. There would also need to be unit tests for key peices
of functionality added along the way, especially for security critical tasks
like TLS certificate validation.

It should go without saying, but just in case, any conversion of chardev
and migration code would have to meet feature parity and especially
performance parity for migration when compared like-for-like features.
eg we should not regress in performance of uncrypted migration vs the
current impl, but encrypted migration might be slower for obvious
reasons (depends on level of encryption alg offload to the CPU)

Regards,
Daniel

[1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsocket.h
[2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsaslcontext.h
[3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnettlscontext.h
[4] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsshsession.h
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 12:32, Daniel P. Berrange wrote:
> So my idea would be that we define a QEMUChannel object and set of APIs to
> standardize all interaction with sockets, pipes, RDMA, whatever $channel,
> and then convert the QEMU features I've mentioned over to use that. I think
> that would be simpler than trying to untangle QEMUFile code from migration
> and then extend its features.

Could it be GIOChannel simply?

1) Chardev is already mostly a wrapper around GIOChannel

2) NBD and VNC could be converted to GIOChannel with relative ease

3) migration is more complicated because (unlike everything else) it
uses a separate thread and blocking sockets, but you could probably
write a GIOChannel-based implementation of QEMUFile.

I found a GIOChannel wrapper for gnutls at
https://github.com/aldebaran/connman/blob/master/gweb/giognutls.c.  It's
not the right license for QEMU (GPLv2-only) but it's only 400 lines of
code.  If necessary I can help with clean-room reverse engineering.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 01:43:12PM +0100, Paolo Bonzini wrote:
> 
> 
> On 04/02/2015 12:32, Daniel P. Berrange wrote:
> > So my idea would be that we define a QEMUChannel object and set of APIs to
> > standardize all interaction with sockets, pipes, RDMA, whatever $channel,
> > and then convert the QEMU features I've mentioned over to use that. I think
> > that would be simpler than trying to untangle QEMUFile code from migration
> > and then extend its features.
> 
> Could it be GIOChannel simply?
> 
> 1) Chardev is already mostly a wrapper around GIOChannel
> 
> 2) NBD and VNC could be converted to GIOChannel with relative ease
> 
> 3) migration is more complicated because (unlike everything else) it
> uses a separate thread and blocking sockets, but you could probably
> write a GIOChannel-based implementation of QEMUFile.

It might be possible to base it on GIOChannel, but IIRC some of the
migration code was using iovecs for I/O and GIOChannel API doesn't
allow for that. So you'd have to sacrifice performance by issuing a
separate syscall for each iovec element which seems sucky to me.
If you think that's an acceptable limitation though, I could certainly
explore use of GIOChannel.

More broadly speaking GIOChannel has fallen out of favour in the
glib ecosystem, with most apps/libraries more focused on use of
the GIO APIs instead, but IIUC QEMU avoids use of the GIO library
due to need to support older glib versions.

> I found a GIOChannel wrapper for gnutls at
> https://github.com/aldebaran/connman/blob/master/gweb/giognutls.c.  It's
> not the right license for QEMU (GPLv2-only) but it's only 400 lines of
> code.  If necessary I can help with clean-room reverse engineering.

It doesn't seem todo any thing related to certificate validation which
explains why it is so short compared ot the gnutls code we already have
for VNC in QEMU. So I don't think it's particularly useful in terms of
saving effort.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Dr. David Alan Gilbert]

* Daniel P. Berrange (berrange@redhat.com) wrote:
> In QEMU there are a number of features which involve communication with an
> external system over an I/O channel of some form. The features include
> migration, NBD, VNC and character devices. The I/O channel in question might
> might be a FIFO pipe, a PTY, a TCP socket, a UNIX domain socket, RMDA channel
> or something else, while the external system can be another QEMU, libvirt, an
> NBD server, or something else.

> Currently the only place where there is any meaningful level of security is
> the VNC server, which supports the VeNCrypt extension providing TLS sessions
> for data encryption and x509 cert validation for authentication, and the SASL
> extension which also provides either encryption of authentication or both.
> The migration data channel is more or less completely unprotected unless it
> is tunnelled via libvirt or equivalent external secure channel. The same is
> true for NBD, though there was a recent discussion about defining an extension
> to use TLS. Likewise serial ports, parallel ports, virtio consoles all use the
> chardev backends which offer no security features.

I think fixing this for migration might simplify stuff for users a lot as well;
the choice of whether libvirt does tunneling or not and what it means
for how block migration happens etc can get very confusing.

> There are some other network related pieces in QEMU, namely the built-in iSCSI,
> RBD, NFS clients, and SPICE server. Since those are all implemented via 3rd
> party libraries rather than natively by QEMU I'm going ignore them for the sake
> of this discussion and focus on network interaction directly implemented in
> QEMU.
> 
> We have a broad goal in OpenStack that every network channel in use must have
> encryption and authentication capabilities. Currently all the communication
> channels between the end user and the cloud infrastructure edge servers are
> secured, but internally a number of the cloud infrastructure components are
> unsecured. For example, we recommend to tunnel migration via libvirt, though
> that excludes use of the NBD for block migration since libvirt can't currently
> tunnel that. Internally we will shortly use  VeNCrypt for protecting VNC
> between the compute hosts and the openstack console proxy edge servers. We
> are lacking the abilty to secure serial ports between the compute nods and
> console proxy.
> 
> Essentially the project considers that it is no longer sufficient to consider
> the private management LAN (on which the cloud infrastructure is deployed) to
> be fully trusted; it must be considered hostile.
> 
> So back to QEMU/KVM, we want to have
> 
>  - Native support for encryption & authentication with migration, since
>    tunnelling via libvirt has a non-negligble performance overhead adding
>    both latency and CPU load
> 
>  - Native support for encryption & authentication with NBD server to enable
>    security of the block migration service
> 
>  - Native support for encryption & authentication with chardev TCP backends
>    to enable security of the serial port consoles.
> 
> As a starting point, the desire is to have TLS for session encryption and
> x509 certificate verification for authentication, but at a later date we'd
> also like to add the ability to use SASL for either aspect too. Thinking about
> QEMU users in general, it would probably be useful if any impl could allow for
> future enhancements such as SSH tunnelling too.
> 
> I have some development cycles available to work on addressing these gaps in
> QEMU, along with relevant experiance since I did the previous VNC server work
> for adding TLS and SASL in QEMU, along with similar in libvirt and GTK-VNC.
> 
> Having looked at the QEMU code for VNC, migration and chardevs I think the
> main stumbling block is currently that QEMU does not have any kind of standard
> approach for dealing with I/O channels internally. Migration has the QEMUFile
> abstraction, but it is currently fairly coupled to the migration code itself
> and limited in scope, because it doesn't actually deal with socket listen
> or connect tasks. That's still part of the migration code itself, QEMUFile
> only deals with I/O transfer, so it is a pretty incomplete abstraction layer.
> The chardev code has a backend abstraction, but that is really horribly
> entwined with the chardev code so not reusable in any way without a rewrite
> from scratch IMHO. The VNC server has alot of useful code for dealing with
> TLS & SASL, but it is somewhat entwined with the VNC server. The VNC server
> doesn't use any I/O abstraction just preferring raw FDs / sockets. I've not
> looked at the NBD code in detail, but I'm presuming it is using raw FDs /
> sockets.
> 
> Since this TLS/SASL code is non-trivial (and obviously security critical), I
> really don't want to end up with 4 separate places to implement it in QEMU.
> IMHO the only practical / sensible approach is to define some kind of standard
> I/O channel API internally to QEMU which migration, NBD, chardev and VNC all
> use. That gives us a single place to integrate all the security mechanisms
> we need to support.  In libvirt we did something like this a little while
> ago by defining a standard internal sockets API[1], with plugins for things
> like SASL[2], TLS[4] and SSH[5] and it has been very successful in simplifying
> the code by centralizing the hairy logic, though I wouldn't aim for exactly
> the same design if doing it again - a more layered approach like QEMU blockdev
> drivers is probably better in retrospect.
> 
> So my idea would be that we define a QEMUChannel object and set of APIs to
> standardize all interaction with sockets, pipes, RDMA, whatever $channel,

I'm not sure if it makes sense for RDMA; it already has a couple of hooks
that go around the back of QEMUFile in migration, and it's transferring the
data via DMA so the page data doesn't go through the same path.

> and then convert the QEMU features I've mentioned over to use that. I think
> that would be simpler than trying to untangle QEMUFile code from migration
> and then extend its features.
> 
> A rough plan of attack would be to split the work in a number of distinct
> pieces
> 
>  - Define a basic QEMUChannel object & APIs.
> 
>  - Convert chardev backend to the QEMUChannel API
> 
>  - Convert migration to the QEMUChannel API
> 
>  - Convert NBD to the QEMUChannel API
> 
>  - Introduce support for TLS to the QEMUChannel object
> 
>  - Introduce support for SASL to the QEMUChannel object
> 
>  - Convert VNC server to the QEMUChannel API
> 
>  - Integrate TLS in migration  (eg flags turn it on/off via CLI/monitor)
> 
>  - Integrate TLS in NBD (eg flags to turn it on/off via CLI/monitor)
> 
>  - Integrate TLS in chardev backend  (eg flags to turn it on/off via CLI/monitor)
> 
> Aside from the first task, I'm not sure that's the best/right order to do
> the work, it is merely the rough set of pieces I see as required. I've also
> no idea how long this would take. It is clearly a pretty large task given
> the different areas of code it touchs.
> 
> In terms of openstack priorities though, migration is the most important,
> which would be tackling migration + NBD, with chardevs being a secondary
> importance. Converting VNC is obviously lowest priority since that already
> has security - it is just a sanity thing to avoid having two inmpls of TLS
> in QEMU long term. There would also need to be unit tests for key peices
> of functionality added along the way, especially for security critical tasks
> like TLS certificate validation.
> 
> It should go without saying, but just in case, any conversion of chardev
> and migration code would have to meet feature parity and especially
> performance parity for migration when compared like-for-like features.
> eg we should not regress in performance of uncrypted migration vs the
> current impl, but encrypted migration might be slower for obvious
> reasons (depends on level of encryption alg offload to the CPU)

Some things to keep in mind:
  1) The current patch from Liang Li doing multi threaded compression
     It just strikes me as an exmaple of another type of filter in the migration
     stream.

  2) Postcopy and fault tolerance need a bidirectional channel; and that back
     channel tends to be small messages.
  3) I'd considered making a separate socket/fd for passing page data
     in the hope of maybe making that page take data via splice; but am not
     sure yet.

Thanks for looking at this,

Dave

> Regards,
> Daniel
> 
> [1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsocket.h
> [2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsaslcontext.h
> [3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnettlscontext.h
> [4] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsshsession.h
> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 14:00, Daniel P. Berrange wrote:
> On Wed, Feb 04, 2015 at 01:43:12PM +0100, Paolo Bonzini wrote:
>>
>>
>> On 04/02/2015 12:32, Daniel P. Berrange wrote:
>>> So my idea would be that we define a QEMUChannel object and set of APIs to
>>> standardize all interaction with sockets, pipes, RDMA, whatever $channel,
>>> and then convert the QEMU features I've mentioned over to use that. I think
>>> that would be simpler than trying to untangle QEMUFile code from migration
>>> and then extend its features.
>>
>> Could it be GIOChannel simply?
>>
>> 1) Chardev is already mostly a wrapper around GIOChannel
>>
>> 2) NBD and VNC could be converted to GIOChannel with relative ease
>>
>> 3) migration is more complicated because (unlike everything else) it
>> uses a separate thread and blocking sockets, but you could probably
>> write a GIOChannel-based implementation of QEMUFile.
> 
> It might be possible to base it on GIOChannel, but IIRC some of the
> migration code was using iovecs for I/O and GIOChannel API doesn't
> allow for that. So you'd have to sacrifice performance by issuing a
> separate syscall for each iovec element which seems sucky to me.
> If you think that's an acceptable limitation though, I could certainly
> explore use of GIOChannel.

As long as QEMUFile remains there and GIOChannel is used only when
encryption is required, that would be an acceptable limitation.  As I
wrote above, migration is a bit special anyway.

> More broadly speaking GIOChannel has fallen out of favour in the
> glib ecosystem, with most apps/libraries more focused on use of
> the GIO APIs instead, but IIUC QEMU avoids use of the GIO library
> due to need to support older glib versions.

Besides that, QEMU developers are not extremely familiar with all the
glib stuff, and GIOChannel is a thin-enough wrapper that it's pretty
easy to understand what's going on.  But that can change if the
alternative has advantages.  Perhaps we could start by converting
chardevs from GIOChannel to GIO.

GIO has TLS bindings (not SASL I think?) in GIO 2.28.  Currently we
require glib 2.12 (released 2006) on POSIX systems and glib 2.20
(released 2009) on Windows.  That's very conservative indeed, I wouldn't
mind changing to a newer version.

>> I found a GIOChannel wrapper for gnutls at
>> https://github.com/aldebaran/connman/blob/master/gweb/giognutls.c.  It's
>> not the right license for QEMU (GPLv2-only) but it's only 400 lines of
>> code.  If necessary I can help with clean-room reverse engineering.
> 
> It doesn't seem todo any thing related to certificate validation which
> explains why it is so short compared ot the gnutls code we already have
> for VNC in QEMU. So I don't think it's particularly useful in terms of
> saving effort.

Yeah, it was only interesting for the GIOChannel boilerplate.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Markus Armbruster]

"Daniel P. Berrange" <berrange@redhat.com> writes:

> On Wed, Feb 04, 2015 at 01:43:12PM +0100, Paolo Bonzini wrote:
>> 
>> 
>> On 04/02/2015 12:32, Daniel P. Berrange wrote:
>> > So my idea would be that we define a QEMUChannel object and set of APIs to
>> > standardize all interaction with sockets, pipes, RDMA, whatever $channel,
>> > and then convert the QEMU features I've mentioned over to use that. I think
>> > that would be simpler than trying to untangle QEMUFile code from migration
>> > and then extend its features.
>> 
>> Could it be GIOChannel simply?
>> 
>> 1) Chardev is already mostly a wrapper around GIOChannel
>> 
>> 2) NBD and VNC could be converted to GIOChannel with relative ease
>> 
>> 3) migration is more complicated because (unlike everything else) it
>> uses a separate thread and blocking sockets, but you could probably
>> write a GIOChannel-based implementation of QEMUFile.
>
> It might be possible to base it on GIOChannel, but IIRC some of the
> migration code was using iovecs for I/O and GIOChannel API doesn't
> allow for that. So you'd have to sacrifice performance by issuing a
> separate syscall for each iovec element which seems sucky to me.
> If you think that's an acceptable limitation though, I could certainly
> explore use of GIOChannel.
>
> More broadly speaking GIOChannel has fallen out of favour in the
> glib ecosystem, with most apps/libraries more focused on use of
> the GIO APIs instead, but IIUC QEMU avoids use of the GIO library
> due to need to support older glib versions.

Remind me: what GLib version are we targeting, and why?

[...]

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Peter Maydell]

On 4 February 2015 at 13:49, Markus Armbruster <armbru@redhat.com> wrote:
> Remind me: what GLib version are we targeting, and why?

Our current minimum is 2.12 (or 2.20 in Windows specific code),
and the reason is RHEL5/Centos 5.

-- PMM

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 01:08:21PM +0000, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrange (berrange@redhat.com) wrote:
> > In QEMU there are a number of features which involve communication with an
> > external system over an I/O channel of some form. The features include
> > migration, NBD, VNC and character devices. The I/O channel in question might
> > might be a FIFO pipe, a PTY, a TCP socket, a UNIX domain socket, RMDA channel
> > or something else, while the external system can be another QEMU, libvirt, an
> > NBD server, or something else.
> 
> > Currently the only place where there is any meaningful level of security is
> > the VNC server, which supports the VeNCrypt extension providing TLS sessions
> > for data encryption and x509 cert validation for authentication, and the SASL
> > extension which also provides either encryption of authentication or both.
> > The migration data channel is more or less completely unprotected unless it
> > is tunnelled via libvirt or equivalent external secure channel. The same is
> > true for NBD, though there was a recent discussion about defining an extension
> > to use TLS. Likewise serial ports, parallel ports, virtio consoles all use the
> > chardev backends which offer no security features.
> 
> I think fixing this for migration might simplify stuff for users a lot as well;
> the choice of whether libvirt does tunneling or not and what it means
> for how block migration happens etc can get very confusing.

Yes, and not helped by the fact that no single current impl offers all
desired features - they are all partially overlapping sets :-(


> > Since this TLS/SASL code is non-trivial (and obviously security critical), I
> > really don't want to end up with 4 separate places to implement it in QEMU.
> > IMHO the only practical / sensible approach is to define some kind of standard
> > I/O channel API internally to QEMU which migration, NBD, chardev and VNC all
> > use. That gives us a single place to integrate all the security mechanisms
> > we need to support.  In libvirt we did something like this a little while
> > ago by defining a standard internal sockets API[1], with plugins for things
> > like SASL[2], TLS[4] and SSH[5] and it has been very successful in simplifying
> > the code by centralizing the hairy logic, though I wouldn't aim for exactly
> > the same design if doing it again - a more layered approach like QEMU blockdev
> > drivers is probably better in retrospect.
> > 
> > So my idea would be that we define a QEMUChannel object and set of APIs to
> > standardize all interaction with sockets, pipes, RDMA, whatever $channel,
> 
> I'm not sure if it makes sense for RDMA; it already has a couple of hooks
> that go around the back of QEMUFile in migration, and it's transferring the
> data via DMA so the page data doesn't go through the same path.

Could you ever anticipate any need for authentication or encryption in
the RDMA based channel ? I don't know enough about RDMA myself to know
if it makes sense or not, other than the fact that any channel between
two separate hosts needs security at some level in the stack.


> Some things to keep in mind:
>   1) The current patch from Liang Li doing multi threaded compression
>      It just strikes me as an exmaple of another type of filter in the migration
>      stream.

Yes, that does make sense in that way,

>   2) Postcopy and fault tolerance need a bidirectional channel; and that back
>      channel tends to be small messages.

TLS will require bidirectional channels too in order to perform the
handshake. SASL will require bidirectional channels too. So this
seems rather inevitable.

>   3) I'd considered making a separate socket/fd for passing page data
>      in the hope of maybe making that page take data via splice; but am not
>      sure yet.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 02:42:20PM +0100, Paolo Bonzini wrote:
> 
> 
> On 04/02/2015 14:00, Daniel P. Berrange wrote:
> > On Wed, Feb 04, 2015 at 01:43:12PM +0100, Paolo Bonzini wrote:
> >>
> >>
> >> On 04/02/2015 12:32, Daniel P. Berrange wrote:
> >>> So my idea would be that we define a QEMUChannel object and set of APIs to
> >>> standardize all interaction with sockets, pipes, RDMA, whatever $channel,
> >>> and then convert the QEMU features I've mentioned over to use that. I think
> >>> that would be simpler than trying to untangle QEMUFile code from migration
> >>> and then extend its features.
> >>
> >> Could it be GIOChannel simply?
> >>
> >> 1) Chardev is already mostly a wrapper around GIOChannel
> >>
> >> 2) NBD and VNC could be converted to GIOChannel with relative ease
> >>
> >> 3) migration is more complicated because (unlike everything else) it
> >> uses a separate thread and blocking sockets, but you could probably
> >> write a GIOChannel-based implementation of QEMUFile.
> > 
> > It might be possible to base it on GIOChannel, but IIRC some of the
> > migration code was using iovecs for I/O and GIOChannel API doesn't
> > allow for that. So you'd have to sacrifice performance by issuing a
> > separate syscall for each iovec element which seems sucky to me.
> > If you think that's an acceptable limitation though, I could certainly
> > explore use of GIOChannel.
> 
> As long as QEMUFile remains there and GIOChannel is used only when
> encryption is required, that would be an acceptable limitation.  As I
> wrote above, migration is a bit special anyway.

I'm not sure I'd like the idea of having different codepaths for
the encrypted vs non-encrypted impl. it seems like a recipe for
increased maintainence work and inconsistent behaviour over the
long term. My thought was that QEMUFile would basically go
away entirely by the end of the conversion, or at most be dealing
with the data rate throttling if that didn't fit nicely into the
generic IO layer.

> > More broadly speaking GIOChannel has fallen out of favour in the
> > glib ecosystem, with most apps/libraries more focused on use of
> > the GIO APIs instead, but IIUC QEMU avoids use of the GIO library
> > due to need to support older glib versions.
> 
> Besides that, QEMU developers are not extremely familiar with all the
> glib stuff, and GIOChannel is a thin-enough wrapper that it's pretty
> easy to understand what's going on.  But that can change if the
> alternative has advantages.  Perhaps we could start by converting
> chardevs from GIOChannel to GIO.
> 
> GIO has TLS bindings (not SASL I think?) in GIO 2.28.  Currently we
> require glib 2.12 (released 2006) on POSIX systems and glib 2.20
> (released 2009) on Windows.  That's very conservative indeed, I wouldn't
> mind changing to a newer version.

Yeah, it has some level of functionality for TLS, but I'm not sure about
the full extent of it and whether it'd be sufficient for what we need
in VNC for example.

The main difference between GIO's APIs and GIOChannel is that the new
GIO stuff is really designed around the idea of asynchronous callbacks
for completion of IO.

  eg

     g_input_stream_read_async(stream, buffer, size, read_done_callback);

 and then when read_done_callback gets triggered you have to call

     g_input_stream_read_finish(stream)

in order to get the success/failure status of the read, and the byte
count. While it is quite nice for new code IME, this is probably quite
alot harder to refit into existing QEMU codebase.  So either GIOChannel
or something custom that is similar to GIOChannel would likely be an
easier fit.

> >> I found a GIOChannel wrapper for gnutls at
> >> https://github.com/aldebaran/connman/blob/master/gweb/giognutls.c.  It's
> >> not the right license for QEMU (GPLv2-only) but it's only 400 lines of
> >> code.  If necessary I can help with clean-room reverse engineering.
> > 
> > It doesn't seem todo any thing related to certificate validation which
> > explains why it is so short compared ot the gnutls code we already have
> > for VNC in QEMU. So I don't think it's particularly useful in terms of
> > saving effort.
> 
> Yeah, it was only interesting for the GIOChannel boilerplate.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 15:08, Daniel P. Berrange wrote:
>> > As long as QEMUFile remains there and GIOChannel is used only when
>> > encryption is required, that would be an acceptable limitation.  As I
>> > wrote above, migration is a bit special anyway.
> I'm not sure I'd like the idea of having different codepaths for
> the encrypted vs non-encrypted impl. it seems like a recipe for
> increased maintainence work and inconsistent behaviour over the
> long term. My thought was that QEMUFile would basically go
> away entirely by the end of the conversion, or at most be dealing
> with the data rate throttling if that didn't fit nicely into the
> generic IO layer.

QEMUFile has a bunch of hooks for RDMA (they were also used by the
never-upstreamed patches to speed up AF_UNIX migration with vmsplice),
so it cannot go away:

typedef struct QEMUFileOps {
    QEMUFilePutBufferFunc *put_buffer;
    QEMUFileGetBufferFunc *get_buffer;
    QEMUFileCloseFunc *close;
    QEMUFileGetFD *get_fd;
    QEMUFileWritevBufferFunc *writev_buffer;
    QEMURamHookFunc *before_ram_iterate;
    QEMURamHookFunc *after_ram_iterate;
    QEMURamHookFunc *hook_ram_load;
    QEMURamSaveFunc *save_page;
    QEMUFileShutdownFunc *shut_down;
} QEMUFileOps;

GIO doesn't provide writev either, so it's not a good match for
non-encrypted migration, which really tries hard to do no copies in
userspace.

> > GIO has TLS bindings (not SASL I think?) in GIO 2.28.  Currently we
> > require glib 2.12 (released 2006) on POSIX systems and glib 2.20
> > (released 2009) on Windows.  That's very conservative indeed, I wouldn't
> > mind changing to a newer version.
>
> Yeah, it has some level of functionality for TLS, but I'm not sure about
> the full extent of it and whether it'd be sufficient for what we need
> in VNC for example.

Okay...  I don't know much about this.

> The main difference between GIO's APIs and GIOChannel is that the new
> GIO stuff is really designed around the idea of asynchronous callbacks
> for completion of IO.
> 
>   eg
> 
>      g_input_stream_read_async(stream, buffer, size, read_done_callback);
> 
>  and then when read_done_callback gets triggered you have to call
> 
>      g_input_stream_read_finish(stream)
> 
> in order to get the success/failure status of the read, and the byte
> count. While it is quite nice for new code IME, this is probably quite
> alot harder to refit into existing QEMU codebase.

It also supports GIOChannel's GSource model via
GPollableInputStream/GPollableOutputStream.  The GNUTLS bindings support
that interface too.

It also supports blocking operation, which is what migration wants.

So I think we could take a look at GIO if its TLS support is advanced
enough for your purposes.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 14:08, Dr. David Alan Gilbert wrote:
>   3) I'd considered making a separate socket/fd for passing page data
>      in the hope of maybe making that page take data via splice; but am not
>      sure yet.

There were even patches for this:
http://lists.gnu.org/archive/html/qemu-devel/2013-11/msg04006.html

The submitter disappeared together with Anthony, but the patches were
pretty good and reused some of the RDMA hooks.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 15:02, Daniel P. Berrange wrote:
> > I'm not sure if it makes sense for RDMA; it already has a couple of hooks
> > that go around the back of QEMUFile in migration, and it's transferring the
> > data via DMA so the page data doesn't go through the same path.
> 
> Could you ever anticipate any need for authentication or encryption in
> the RDMA based channel ? I don't know enough about RDMA myself to know
> if it makes sense or not, other than the fact that any channel between
> two separate hosts needs security at some level in the stack.

Authentication, possibly; but I don't think encryption makes sense.  Marcel?

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 03:23:22PM +0100, Paolo Bonzini wrote:
> 
> 
> On 04/02/2015 15:08, Daniel P. Berrange wrote:
> >> > As long as QEMUFile remains there and GIOChannel is used only when
> >> > encryption is required, that would be an acceptable limitation.  As I
> >> > wrote above, migration is a bit special anyway.
> > I'm not sure I'd like the idea of having different codepaths for
> > the encrypted vs non-encrypted impl. it seems like a recipe for
> > increased maintainence work and inconsistent behaviour over the
> > long term. My thought was that QEMUFile would basically go
> > away entirely by the end of the conversion, or at most be dealing
> > with the data rate throttling if that didn't fit nicely into the
> > generic IO layer.
> 
> QEMUFile has a bunch of hooks for RDMA (they were also used by the
> never-upstreamed patches to speed up AF_UNIX migration with vmsplice),
> so it cannot go away:
> 
> typedef struct QEMUFileOps {
>     QEMUFilePutBufferFunc *put_buffer;
>     QEMUFileGetBufferFunc *get_buffer;
>     QEMUFileCloseFunc *close;
>     QEMUFileGetFD *get_fd;
>     QEMUFileWritevBufferFunc *writev_buffer;
>     QEMURamHookFunc *before_ram_iterate;
>     QEMURamHookFunc *after_ram_iterate;
>     QEMURamHookFunc *hook_ram_load;
>     QEMURamSaveFunc *save_page;
>     QEMUFileShutdownFunc *shut_down;
> } QEMUFileOps;
> 
> GIO doesn't provide writev either, so it's not a good match for
> non-encrypted migration, which really tries hard to do no copies in
> userspace.

Ok, maybe RDMA will still need QEMUFile, but for non-encrypted TCP
I'd hope to be able to achieve zero-copy with the new API too - it
would certainly be my intention/goal.

> > The main difference between GIO's APIs and GIOChannel is that the new
> > GIO stuff is really designed around the idea of asynchronous callbacks
> > for completion of IO.
> > 
> >   eg
> > 
> >      g_input_stream_read_async(stream, buffer, size, read_done_callback);
> > 
> >  and then when read_done_callback gets triggered you have to call
> > 
> >      g_input_stream_read_finish(stream)
> > 
> > in order to get the success/failure status of the read, and the byte
> > count. While it is quite nice for new code IME, this is probably quite
> > alot harder to refit into existing QEMU codebase.
> 
> It also supports GIOChannel's GSource model via
> GPollableInputStream/GPollableOutputStream.  The GNUTLS bindings support
> that interface too.
> 
> It also supports blocking operation, which is what migration wants.
> 
> So I think we could take a look at GIO if its TLS support is advanced
> enough for your purposes.

Ok, I'll investigate GIO a little further to see how practical a fit it
is for us.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Dr. David Alan Gilbert]

* Paolo Bonzini (pbonzini@redhat.com) wrote:
> 
> 
> On 04/02/2015 14:08, Dr. David Alan Gilbert wrote:
> >   3) I'd considered making a separate socket/fd for passing page data
> >      in the hope of maybe making that page take data via splice; but am not
> >      sure yet.
> 
> There were even patches for this:
> http://lists.gnu.org/archive/html/qemu-devel/2013-11/msg04006.html
> 
> The submitter disappeared together with Anthony, but the patches were
> pretty good and reused some of the RDMA hooks.

Yep, I looked at those a while back and intend to have another look
at some time; I think they were primarily intended for same-host migration
to upgrade running QEMU instances.

Dave

> 
> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Marcel Apfelbaum]

On 02/04/2015 04:28 PM, Paolo Bonzini wrote:
>
>
> On 04/02/2015 15:02, Daniel P. Berrange wrote:
>>> I'm not sure if it makes sense for RDMA; it already has a couple of hooks
>>> that go around the back of QEMUFile in migration, and it's transferring the
>>> data via DMA so the page data doesn't go through the same path.
>>
>> Could you ever anticipate any need for authentication or encryption in
>> the RDMA based channel ? I don't know enough about RDMA myself to know
>> if it makes sense or not, other than the fact that any channel between
>> two separate hosts needs security at some level in the stack.
>
> Authentication, possibly; but I don't think encryption makes sense.  Marcel?
I personally think that the protocol is safe enough, but as always there are holes
and I am not a security expert:

     "RDMA mechanisms can create a potential security vulnerability. A node may access another nodes
      memory region that was supposed to be banned.
      In order to protect remote memory access to unauthorized memory areas, InfiniBand defines memory
      protection mechanisms, where a remote memory access requires a special key (R_Key). The R_Key is
      negotiated between the peers and is validated at the target’s system HCA card. In case of illegal key the
      packet is dropped. The R_Key requirement is built into silicon and driver code and cannot be disabled
      even when attacker compromises root/admin/superuser account on one or multiple servers."

More on Layer 2 attacks and how Infiniband handle those:
     http://www.mellanox.com/related-docs/whitepapers/WP_Secuirty_In_InfiniBand_Fabrics_Final.pdf

I hope I helped,
Marcel

>
> Paolo
>

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 04:48:44PM +0200, Marcel Apfelbaum wrote:
> On 02/04/2015 04:28 PM, Paolo Bonzini wrote:
> >
> >
> >On 04/02/2015 15:02, Daniel P. Berrange wrote:
> >>>I'm not sure if it makes sense for RDMA; it already has a couple of hooks
> >>>that go around the back of QEMUFile in migration, and it's transferring the
> >>>data via DMA so the page data doesn't go through the same path.
> >>
> >>Could you ever anticipate any need for authentication or encryption in
> >>the RDMA based channel ? I don't know enough about RDMA myself to know
> >>if it makes sense or not, other than the fact that any channel between
> >>two separate hosts needs security at some level in the stack.
> >
> >Authentication, possibly; but I don't think encryption makes sense.  Marcel?
> I personally think that the protocol is safe enough, but as always there are holes
> and I am not a security expert:
> 
>     "RDMA mechanisms can create a potential security vulnerability. A node may access another nodes
>      memory region that was supposed to be banned.
>      In order to protect remote memory access to unauthorized memory areas, InfiniBand defines memory
>      protection mechanisms, where a remote memory access requires a special key (R_Key). The R_Key is
>      negotiated between the peers and is validated at the target’s system HCA card. In case of illegal key the
>      packet is dropped. The R_Key requirement is built into silicon and driver code and cannot be disabled
>      even when attacker compromises root/admin/superuser account on one or multiple servers."
> 
> More on Layer 2 attacks and how Infiniband handle those:
>     http://www.mellanox.com/related-docs/whitepapers/WP_Secuirty_In_InfiniBand_Fabrics_Final.pdf
>

Ok I guess we could probably just assume RDMA is sufficient as is for now.
We can fairly easily revisit it later if we find it it needs more work

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 15:34, Daniel P. Berrange wrote:
> > GIO doesn't provide writev either, so it's not a good match for
> > non-encrypted migration, which really tries hard to do no copies in
> > userspace.
> 
> Ok, maybe RDMA will still need QEMUFile, but for non-encrypted TCP
> I'd hope to be able to achieve zero-copy with the new API too - it
> would certainly be my intention/goal.

For GIO/GIOChannel, you'd have to choose between zerocopy and many
syscalls, or one copy and few syscalls.  Since every page has two iov
entries, one of which is just 8 bytes, one copy and few syscalls is
probably faster---but still only half the speed of writev.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 04:04:33PM +0100, Paolo Bonzini wrote:
> 
> 
> On 04/02/2015 15:34, Daniel P. Berrange wrote:
> > > GIO doesn't provide writev either, so it's not a good match for
> > > non-encrypted migration, which really tries hard to do no copies in
> > > userspace.
> > 
> > Ok, maybe RDMA will still need QEMUFile, but for non-encrypted TCP
> > I'd hope to be able to achieve zero-copy with the new API too - it
> > would certainly be my intention/goal.
> 
> For GIO/GIOChannel, you'd have to choose between zerocopy and many
> syscalls, or one copy and few syscalls.  Since every page has two iov
> entries, one of which is just 8 bytes, one copy and few syscalls is
> probably faster---but still only half the speed of writev.

That could be an argument in favour of defining a QEMUIOChannel
instead. The use of GIOChannel is only compelling if we gain some
significant benefit from using the standard glib API in this scenario
and I'm not clear that we really do, other than developer familiarity
with glib. GIO could be compelling if we could leverage its TLS
integration

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 16:11, Daniel P. Berrange wrote:
> > For GIO/GIOChannel, you'd have to choose between zerocopy and many
> > syscalls, or one copy and few syscalls.  Since every page has two iov
> > entries, one of which is just 8 bytes, one copy and few syscalls is
> > probably faster---but still only half the speed of writev.
>
> That could be an argument in favour of defining a QEMUIOChannel
> instead. The use of GIOChannel is only compelling if we gain some
> significant benefit from using the standard glib API in this scenario
> and I'm not clear that we really do, other than developer familiarity
> with glib.

GIOChannel's advantage is that---even though it may not be used for
regular migration---integration with chardev would be really easy.

> GIO could be compelling if we could leverage its TLS integration

I agree.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 04:22:26PM +0100, Paolo Bonzini wrote:
> 
> 
> On 04/02/2015 16:11, Daniel P. Berrange wrote:
> > > For GIO/GIOChannel, you'd have to choose between zerocopy and many
> > > syscalls, or one copy and few syscalls.  Since every page has two iov
> > > entries, one of which is just 8 bytes, one copy and few syscalls is
> > > probably faster---but still only half the speed of writev.
> >
> > That could be an argument in favour of defining a QEMUIOChannel
> > instead. The use of GIOChannel is only compelling if we gain some
> > significant benefit from using the standard glib API in this scenario
> > and I'm not clear that we really do, other than developer familiarity
> > with glib.
> 
> GIOChannel's advantage is that---even though it may not be used for
> regular migration---integration with chardev would be really easy.

If we did a QEMUIOChannel that was basically the same as IOChannel, but
with support for iovec writev/readv it'd mostly be drop-in replacement
for the chardev code I guess.

> 
> > GIO could be compelling if we could leverage its TLS integration
> 
> I agree.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On 4 February 2015 at 13:49, Markus Armbruster <armbru@redhat.com> wrote:
>> Remind me: what GLib version are we targeting, and why?
>
> Our current minimum is 2.12 (or 2.20 in Windows specific code),
> and the reason is RHEL5/Centos 5.

Any idea when we can move on?

Don't get me started on the wisdom of developing or deploying upstream
QEMU on RHEL-*5*.

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 05:33:36PM +0100, Markus Armbruster wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> > On 4 February 2015 at 13:49, Markus Armbruster <armbru@redhat.com> wrote:
> >> Remind me: what GLib version are we targeting, and why?
> >
> > Our current minimum is 2.12 (or 2.20 in Windows specific code),
> > and the reason is RHEL5/Centos 5.
> 
> Any idea when we can move on?
> 
> Don't get me started on the wisdom of developing or deploying upstream
> QEMU on RHEL-*5*.

If we want to make use of GIO, we would need  2.28 at a bare minimum.
This is present in RHEL-6 vintage onwards.

Unfortunately, if we want to make use of GIO with TLS, then we would
realistically need to have 2.38, since the changelog shows there are
some pretty major bugs in the TLS impl in earlier versions. eg you
could not read and write to a TLS socket at the same time prior
to 2.38 [1] :-( So that version of GIO, with the glib-networking addon
to provide TLS, would mean only running on RHEL-7, which feels like it
is so new it would likely be a showstopper.

Regards,
Daniel

[1] https://git.gnome.org/browse/glib-networking/tree/NEWS#n325
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 16:26, Daniel P. Berrange wrote:
> > GIOChannel's advantage is that---even though it may not be used for
> > regular migration---integration with chardev would be really easy.
> 
> If we did a QEMUIOChannel that was basically the same as IOChannel, but
> with support for iovec writev/readv it'd mostly be drop-in replacement
> for the chardev code I guess.

Alternatively, QEMUIOChannel could be a very thin wrapper around GIOChannel.

If the QEMUIOChannel is created with qemu_io_channel_unix_get_fd, the
QEMUIOChannel could use g_io_channel_unix_get_fd to do writev/readv;
otherwise it would just use compatibility code.

This relieves you from having to rewrite all the Win32 code that glib
already has.

> So that version of GIO, with the glib-networking addon
> to provide TLS, would mean only running on RHEL-7, which feels like it
> is so new it would likely be a showstopper.

As long as the non-TLS stuff works on older distros, that would be okay
for me.  Personally I don't care if VNC-TLS regresses on older distros,
either.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]

On 02/04/2015 07:02 AM, Daniel P. Berrange wrote:

>>
>> I think fixing this for migration might simplify stuff for users a lot as well;
>> the choice of whether libvirt does tunneling or not and what it means
>> for how block migration happens etc can get very confusing.
> 
> Yes, and not helped by the fact that no single current impl offers all
> desired features - they are all partially overlapping sets :-(

>> Some things to keep in mind:
>>   1) The current patch from Liang Li doing multi threaded compression
>>      It just strikes me as an exmaple of another type of filter in the migration
>>      stream.
> 
> Yes, that does make sense in that way,
> 
>>   2) Postcopy and fault tolerance need a bidirectional channel; and that back
>>      channel tends to be small messages.
> 
> TLS will require bidirectional channels too in order to perform the
> handshake. SASL will require bidirectional channels too. So this
> seems rather inevitable.
> 
>>   3) I'd considered making a separate socket/fd for passing page data
>>      in the hope of maybe making that page take data via splice; but am not
>>      sure yet.

Another thing to think about - should migration to file be something
that can be encrypted?  There, you don't have the luxury of a
bidirectional channel, but securing the machine state so that only an
authenticated user can decrypt to reload the state later sounds like
another benefit that might be possible.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Peter Maydell]

On 4 February 2015 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
>> On 4 February 2015 at 13:49, Markus Armbruster <armbru@redhat.com> wrote:
>>> Remind me: what GLib version are we targeting, and why?
>>
>> Our current minimum is 2.12 (or 2.20 in Windows specific code),
>> and the reason is RHEL5/Centos 5.
>
> Any idea when we can move on?
>
> Don't get me started on the wisdom of developing or deploying upstream
> QEMU on RHEL-*5*.

Not all of QEMU's use cases are KVM-using VM deployments, not all
compute cluster deployments are primarily directed to that, and
not all industries rev their supported OS platforms very fast.
For instance the EDA tools industry only added RHEL6 support
in 2012 for new design starts, and given the typical length of
a project it's not that implausible to still have RHEL5.

That said, we don't have to insist on supporting the most
ancient version of everything ever, and now might be a reasonable
time to move forward. I wouldn't want to move further forward
than RHEL6's version, though.

Moving beyond 2.22 would be awkward for me in that my OSX
box only has 2.22 because fink doesn't have anything newer.
I could probably deal with that somehow (switching to some
other package system, probably).

Debian stable is "2.33.12+really2.32.4-5" and oldstable
is "2.24.2-1" (and if my googling is right is an LTS release).

Ubuntu Lucid (LTS release) is 2.24; Precise (also LTS)
is 2.32.

Daniel says RHEL6 has 2.28.

That suggests to me that we could reasonably advance to
2.22 or 2.24 if it seemed beneficial, but not beyond that.
Is there anything particularly worthwhile that would get us?

-- PMM

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Paolo Bonzini]

On 04/02/2015 21:41, Peter Maydell wrote:
> That suggests to me that we could reasonably advance to
> 2.22 or 2.24 if it seemed beneficial, but not beyond that.
> Is there anything particularly worthwhile that would get us?

2.22 is the first version to have GIO.

2.28 would give us TLS support in GIO, but Daniel said it's buggy.

2.32 has new-style synchronization primitives, for which we currently
have some hideous compatibility wrappers.

It looks like there's no particular reason yet to bump our requirement.

Paolo

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On 4 February 2015 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
>> Peter Maydell <peter.maydell@linaro.org> writes:
>>> On 4 February 2015 at 13:49, Markus Armbruster <armbru@redhat.com> wrote:
>>>> Remind me: what GLib version are we targeting, and why?
>>>
>>> Our current minimum is 2.12 (or 2.20 in Windows specific code),
>>> and the reason is RHEL5/Centos 5.
>>
>> Any idea when we can move on?
>>
>> Don't get me started on the wisdom of developing or deploying upstream
>> QEMU on RHEL-*5*.
>
> Not all of QEMU's use cases are KVM-using VM deployments, not all
> compute cluster deployments are primarily directed to that, and
> not all industries rev their supported OS platforms very fast.
> For instance the EDA tools industry only added RHEL6 support
> in 2012 for new design starts, and given the typical length of
> a project it's not that implausible to still have RHEL5.

If you can compile upstream QEMU, compiling GLib shouldn't be an
insurmountable obstacle.

> That said, we don't have to insist on supporting the most
> ancient version of everything ever, and now might be a reasonable
> time to move forward. I wouldn't want to move further forward
> than RHEL6's version, though.

Fair enough.

> Moving beyond 2.22 would be awkward for me in that my OSX
> box only has 2.22 because fink doesn't have anything newer.
> I could probably deal with that somehow (switching to some
> other package system, probably).
>
> Debian stable is "2.33.12+really2.32.4-5" and oldstable
> is "2.24.2-1" (and if my googling is right is an LTS release).
>
> Ubuntu Lucid (LTS release) is 2.24; Precise (also LTS)
> is 2.32.
>
> Daniel says RHEL6 has 2.28.
>
> That suggests to me that we could reasonably advance to
> 2.22 or 2.24 if it seemed beneficial, but not beyond that.
> Is there anything particularly worthwhile that would get us?

"Worthwhile" is in the eye of the beholder.  Chaining ourselves to 7+
year-old libraries means we get to work around annoyances (quick grep:
commit f8833a3 01a2050), we compromise on testing when the libraries are
actually old[*] (commit 9d41401), and certain improvements won't get
done because they're too much of a bother (commit 02c4f26).  These are
all paper cuts.  Is suffering them worthwhile?


[*] Rule of thumb: if you want to run an upstream version of X, run it
under an OS the upstream developers actually use every day, or do your
own testing.

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Dr. David Alan Gilbert]

* Eric Blake (eblake@redhat.com) wrote:
> On 02/04/2015 07:02 AM, Daniel P. Berrange wrote:
> 
> >>
> >> I think fixing this for migration might simplify stuff for users a lot as well;
> >> the choice of whether libvirt does tunneling or not and what it means
> >> for how block migration happens etc can get very confusing.
> > 
> > Yes, and not helped by the fact that no single current impl offers all
> > desired features - they are all partially overlapping sets :-(
> 
> >> Some things to keep in mind:
> >>   1) The current patch from Liang Li doing multi threaded compression
> >>      It just strikes me as an exmaple of another type of filter in the migration
> >>      stream.
> > 
> > Yes, that does make sense in that way,
> > 
> >>   2) Postcopy and fault tolerance need a bidirectional channel; and that back
> >>      channel tends to be small messages.
> > 
> > TLS will require bidirectional channels too in order to perform the
> > handshake. SASL will require bidirectional channels too. So this
> > seems rather inevitable.
> > 
> >>   3) I'd considered making a separate socket/fd for passing page data
> >>      in the hope of maybe making that page take data via splice; but am not
> >>      sure yet.
> 
> Another thing to think about - should migration to file be something
> that can be encrypted?  There, you don't have the luxury of a
> bidirectional channel, but securing the machine state so that only an
> authenticated user can decrypt to reload the state later sounds like
> another benefit that might be possible.

That's something that's pretty easy to do via exec-migration; so I'm
not sure it's worth doing anything particularly special for it.

Dave

> 
> -- 
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
> 


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

On Wed, Feb 04, 2015 at 02:42:20PM +0100, Paolo Bonzini wrote:
> GIO has TLS bindings (not SASL I think?) in GIO 2.28.  Currently we
> require glib 2.12 (released 2006) on POSIX systems and glib 2.20
> (released 2009) on Windows.  That's very conservative indeed, I wouldn't
> mind changing to a newer version.

Yes, the glib 2.12 requirement is too conservative.  We can definitely
move to a higher version.

Red Hat Enterprise Linux 6.6 is on 2.28.
Debian wheezy (stable) is on glib 2.33.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Cornelia Huck]

On Thu, 5 Feb 2015 14:38:36 +0000
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Wed, Feb 04, 2015 at 02:42:20PM +0100, Paolo Bonzini wrote:
> > GIO has TLS bindings (not SASL I think?) in GIO 2.28.  Currently we
> > require glib 2.12 (released 2006) on POSIX systems and glib 2.20
> > (released 2009) on Windows.  That's very conservative indeed, I wouldn't
> > mind changing to a newer version.
> 
> Yes, the glib 2.12 requirement is too conservative.  We can definitely
> move to a higher version.
> 
> Red Hat Enterprise Linux 6.6 is on 2.28.
> Debian wheezy (stable) is on glib 2.33.

SLES 11 SP 3 seems to be on 2.22, so I don't think we can go later than
that yet.

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Peter Maydell]

On 5 February 2015 at 14:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> Yes, the glib 2.12 requirement is too conservative.  We can definitely
> move to a higher version.
>
> Red Hat Enterprise Linux 6.6 is on 2.28.
> Debian wheezy (stable) is on glib 2.33.

I'd rather not drop support for Debian oldstable or Ubuntu LTS
distros, though, so 2.24 (or 2.22 if you want to maintain support
for fink on OSX).

-- PMM

Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Daniel P. Berrange]

On Wed, Feb 04, 2015 at 11:32:29AM +0000, Daniel P. Berrange wrote:
> In QEMU there are a number of features which involve communication with an
> external system over an I/O channel of some form. The features include
> migration, NBD, VNC and character devices. The I/O channel in question might
> might be a FIFO pipe, a PTY, a TCP socket, a UNIX domain socket, RMDA channel
> or something else, while the external system can be another QEMU, libvirt, an
> NBD server, or something else.

[snip]

> So back to QEMU/KVM, we want to have
> 
>  - Native support for encryption & authentication with migration, since
>    tunnelling via libvirt has a non-negligble performance overhead adding
>    both latency and CPU load
> 
>  - Native support for encryption & authentication with NBD server to enable
>    security of the block migration service
> 
>  - Native support for encryption & authentication with chardev TCP backends
>    to enable security of the serial port consoles.
> 
> As a starting point, the desire is to have TLS for session encryption and
> x509 certificate verification for authentication, but at a later date we'd
> also like to add the ability to use SASL for either aspect too. Thinking about
> QEMU users in general, it would probably be useful if any impl could allow for
> future enhancements such as SSH tunnelling too.
> 
> I have some development cycles available to work on addressing these gaps in
> QEMU, along with relevant experiance since I did the previous VNC server work
> for adding TLS and SASL in QEMU, along with similar in libvirt and GTK-VNC.
> 
> Having looked at the QEMU code for VNC, migration and chardevs I think the
> main stumbling block is currently that QEMU does not have any kind of standard
> approach for dealing with I/O channels internally. Migration has the QEMUFile
> abstraction, but it is currently fairly coupled to the migration code itself
> and limited in scope, because it doesn't actually deal with socket listen
> or connect tasks. That's still part of the migration code itself, QEMUFile
> only deals with I/O transfer, so it is a pretty incomplete abstraction layer.
> The chardev code has a backend abstraction, but that is really horribly
> entwined with the chardev code so not reusable in any way without a rewrite
> from scratch IMHO. The VNC server has alot of useful code for dealing with
> TLS & SASL, but it is somewhat entwined with the VNC server. The VNC server
> doesn't use any I/O abstraction just preferring raw FDs / sockets. I've not
> looked at the NBD code in detail, but I'm presuming it is using raw FDs /
> sockets.
> 
> Since this TLS/SASL code is non-trivial (and obviously security critical), I
> really don't want to end up with 4 separate places to implement it in QEMU.
> IMHO the only practical / sensible approach is to define some kind of standard
> I/O channel API internally to QEMU which migration, NBD, chardev and VNC all
> use. That gives us a single place to integrate all the security mechanisms
> we need to support.  In libvirt we did something like this a little while
> ago by defining a standard internal sockets API[1], with plugins for things
> like SASL[2], TLS[4] and SSH[5] and it has been very successful in simplifying
> the code by centralizing the hairy logic, though I wouldn't aim for exactly
> the same design if doing it again - a more layered approach like QEMU blockdev
> drivers is probably better in retrospect.
> 
> So my idea would be that we define a QEMUChannel object and set of APIs to
> standardize all interaction with sockets, pipes, RDMA, whatever $channel,
> and then convert the QEMU features I've mentioned over to use that. I think
> that would be simpler than trying to untangle QEMUFile code from migration
> and then extend its features.
> 
> A rough plan of attack would be to split the work in a number of distinct
> pieces
> 
>  - Define a basic QEMUChannel object & APIs.
> 
>  - Convert chardev backend to the QEMUChannel API
> 
>  - Convert migration to the QEMUChannel API
> 
>  - Convert NBD to the QEMUChannel API
> 
>  - Introduce support for TLS to the QEMUChannel object
> 
>  - Introduce support for SASL to the QEMUChannel object
> 
>  - Convert VNC server to the QEMUChannel API
> 
>  - Integrate TLS in migration  (eg flags turn it on/off via CLI/monitor)
> 
>  - Integrate TLS in NBD (eg flags to turn it on/off via CLI/monitor)
> 
>  - Integrate TLS in chardev backend  (eg flags to turn it on/off via CLI/monitor)
> 
> Aside from the first task, I'm not sure that's the best/right order to do
> the work, it is merely the rough set of pieces I see as required. I've also
> no idea how long this would take. It is clearly a pretty large task given
> the different areas of code it touchs.

FYI, although I've been quiet on the list since this thread, I have
actually been working on this proposal. Since we ruled the use of GIO
due to the v recent min library version it would require, I looked at
the option of using the GIOChannel framework. In the end I decided
that was a dead-end since its API is rather poorly designed for
extensibility. Extending it would be more work than ignoring it,
and be less maintainable due to the need to rely on undocumented
glib hacks. So I've been moving forward with a design that is
inspired by GIOChannel, but using struct iovec to get the efficient
I/O path migration requires, and without all the charset encoding
stuff we have no need for. This has the added benefit that it lets
us fully integrate with QOM Object model and the Error object
error reporting.

So far I've

 - Defined QIOChannel base object interface
 - Implemented a QIOChannelSocket subclass for sockets
 - Implemented a QIOChannelFile subclass for pipes, devices, files, etc
 - Implemented a QIOChannelTLS subclass for the TLS protocol
 - Converted VNC to use QIOChannelSocket, QIOChannelTLS & QIOChannelWebsock
 - Converted chardev to use QIOChannelSocket & QIOChannelFile

Using these new APIs is having a dramatic positive effect on the VNC
server code, allowing all the CONFIG_* conditionals to be removed and
greatly simplifying the code.

My intention is to get the chardev backend converted and supporting TLS
encryption and websockets encapsulation before posting it for formal
review. To get there I need to

 - Finish a QIOChannelWebsock subclass for websockets protocol
 - Finish converting VNC server to new APIs
 - Define some kind of generic '-tls' command line arg for setting
   up TLS x509 credentials to be shared by -chardev & -vnc backends
 - Integrate QIOChannelTLS and QIOChannelWebsock into the chardev
   backends where appropriate (TCP chardev only most likely)
 - Finish Win32 portability of QIOChannelSocket|File subclasses

This will likely be a few weeks out from now at least. If people really
desperately want to see the work-in-progress

   https://github.com/berrange/qemu/commits/qemu-io-channel-4

Bear in mind that git repo is just a daily snapshot which crashes
in several places, doesn't yet compile on Win32, clubs baby seals,
etc, etc.

Once the first stage is posted & positively reviewed, I'll move on
to the migration client & server code and then the NBD client &
server code, converting both to use QIOChannel and thus (trivially)
gain TLS support.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|