[TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[George Dunlap]

* Hardware:

Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz

Not sure exactly what mobo I have, but I get this error if I boot
without any iommu options:

Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata #47, #53

* Software:

Debian Wheezy dom0 (Stock Debian 3.2 kernel)

* Guest operating systems:

Debian Wheezy (Stock debian 3.2 kernel)

* Functionality tested:

xl pci-asssignable-add
xl pci-assignable-remove -r

Booting PV:
- xl pci-attach
- xl pci-remove [fails due to guest kernel bug]
- Shutting down the guest, starting a new one and re-assigning
- pci=[] in config file
- Shutting down guest, starting new one (with config file)

Booting HVM:
 - xl pci-attach,  xl pci-detach, re-attach
 - Shutting down guest with device assigned, starting a new one
 - pci=[] in config file
 - pci-detach after booting with config file

* Comments:

xl pci-detach fails, but this is a known issue that Konrad has fixed
upstream: http://bugs.xenproject.org/xen/bug/12

If the pci-detach fails, then afterwards you can't attach it somewhere
else while the guest is running, but you can re-assign it after the
guest is shut down.

The big problems I see, however, are as follows:

- For PV guests there is no user-visible indication that the IOMMU has
been disabled.  All of the commands work, albeit with slightly
different configurations (e.g., using GSIs instead of MSIs).

- For HVM guests, the only user-visible indication tha the IOMMU has
been disabled is the following error message on the command-line:

# xl pci-attach h0 07:00.0
libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed

However, the device itself ends up passed-through to the guest anyway;
the guest seems to be able to see it and interact with it normally.
This is particularly scary, as in theory this should not be possible
without a working IOMMU.

I don't think this is a blocker for 4.3, but we should definitely
release note it, and for 4.4 add a check to see if there is a
functioning IOMMU and only add a device if there's an override set.

 -George

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Jan Beulich]

>>> On 28.06.13 at 17:37, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> - For HVM guests, the only user-visible indication tha the IOMMU has
> been disabled is the following error message on the command-line:
> 
> # xl pci-attach h0 07:00.0
> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
> 
> However, the device itself ends up passed-through to the guest anyway;
> the guest seems to be able to see it and interact with it normally.
> This is particularly scary, as in theory this should not be possible
> without a working IOMMU.
> 
> I don't think this is a blocker for 4.3, but we should definitely
> release note it, and for 4.4 add a check to see if there is a
> functioning IOMMU and only add a device if there's an override set.

To me this very much looks like a security problem (which I
think we should fix asap).

As I tried this the other day with a boot time assignment, and
it prevented the guest from booting (which is how it should be)
- are you also seeing the guest happily using such device when
assigned via guest config file?

Knowing that may hint at where to look for the actual problem.

Also, I can't really see how the guest would be able to interact
with a half way assigned device properly - I could imagine you
being able to look at its config space, and perhaps load the
driver, but I can't see an I/O to succeed, at least not as long
as any bus mastering is being used the device (this ought to
crash this guest, another guest, or the host, or deliver corrupt
data). Purely port based I/O would likely work, but other than
serial cards I can't think of many things that would do so.

Jan

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[George Dunlap]

On 28/06/13 17:00, Jan Beulich wrote:
>>>> On 28.06.13 at 17:37, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>> - For HVM guests, the only user-visible indication tha the IOMMU has
>> been disabled is the following error message on the command-line:
>>
>> # xl pci-attach h0 07:00.0
>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>
>> However, the device itself ends up passed-through to the guest anyway;
>> the guest seems to be able to see it and interact with it normally.
>> This is particularly scary, as in theory this should not be possible
>> without a working IOMMU.
>>
>> I don't think this is a blocker for 4.3, but we should definitely
>> release note it, and for 4.4 add a check to see if there is a
>> functioning IOMMU and only add a device if there's an override set.
> To me this very much looks like a security problem (which I
> think we should fix asap).
>
> As I tried this the other day with a boot time assignment, and
> it prevented the guest from booting (which is how it should be)
> - are you also seeing the guest happily using such device when
> assigned via guest config file?
>
> Knowing that may hint at where to look for the actual problem.
>
> Also, I can't really see how the guest would be able to interact
> with a half way assigned device properly - I could imagine you
> being able to look at its config space, and perhaps load the
> driver, but I can't see an I/O to succeed, at least not as long
> as any bus mastering is being used the device (this ought to
> crash this guest, another guest, or the host, or deliver corrupt
> data). Purely port based I/O would likely work, but other than
> serial cards I can't think of many things that would do so.

I get basically the same results; adding "pci=['07:00.0']" to the config 
file:

# xl create h0
Parsing config from h0
xc: info: VIRTUAL MEMORY ARRANGEMENT:
   Loader:        0000000000100000->000000000019ee28
   Modules:       0000000000000000->0000000000000000
   TOTAL:         0000000000000000->00000001ff800000
   ENTRY ADDRESS: 0000000000100608
xc: info: PHYSICAL MEMORY ALLOCATION:
   4KB PAGES: 0x0000000000000200
   2MB PAGES: 0x00000000000009fb
   1GB PAGES: 0x0000000000000003
libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed: 
Function not implemented
Daemon running with PID 4346
kodo2:~# xl pci-assignable-list
kodo2:~# xl pci-list h0
Vdev Device
05.0 0000:07:00.0

And in the guest:

# lspci
[snip]
00:05.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network 
Connection (rev 02)
# ifconfig -a
[snip]
eth2      Link encap:Ethernet  HWaddr 00:1b:21:3e:fe:90
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
           Memory:f3200000-f3220000
[snip]
# ifup eth2
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth2/00:1b:21:3e:fe:90
Sending on   LPF/eth2/00:1b:21:3e:fe:90
Sending on   Socket/fallback
DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 8

Since the cable is not plugged in, the device never comes up.  So it's 
quite possible that since it's sending packets but not receiving 
anything, that it's either not doing any DMA, or that it's DMA'ing out 
junk but it doesn't matter.

Either way, if someone *is* passing through a device, it is probably a 
security issue.

  -George

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Ian Campbell]

On Fri, 2013-06-28 at 17:00 +0100, Jan Beulich wrote:
> Also, I can't really see how the guest would be able to interact
> with a half way assigned device properly - I could imagine you
> being able to look at its config space, and perhaps load the
> driver, but I can't see an I/O to succeed, at least not as long
> as any bus mastering is being used the device (this ought to
> crash this guest, another guest, or the host, or deliver corrupt
> data). Purely port based I/O would likely work, but other than
> serial cards I can't think of many things that would do so.

That's what I was thinking. I would be useful to know what sort of
device this is and to what extent it is "working".

Ian.

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[George Dunlap]

On 28/06/13 17:00, Jan Beulich wrote:
>>>> On 28.06.13 at 17:37, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>> - For HVM guests, the only user-visible indication tha the IOMMU has
>> been disabled is the following error message on the command-line:
>>
>> # xl pci-attach h0 07:00.0
>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>
>> However, the device itself ends up passed-through to the guest anyway;
>> the guest seems to be able to see it and interact with it normally.
>> This is particularly scary, as in theory this should not be possible
>> without a working IOMMU.
>>
>> I don't think this is a blocker for 4.3, but we should definitely
>> release note it, and for 4.4 add a check to see if there is a
>> functioning IOMMU and only add a device if there's an override set.
> To me this very much looks like a security problem (which I
> think we should fix asap).

Is it worth delaying the release (yet) another week for?

Probably the simplest solution at the moment, if there's an easy way for 
the toolstack to figure out whether there is a working IOMMU or not, is 
to simply not allow pass-through without an IOMMU unless there is an 
override option.

Thoughts?
  -George

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Jan Beulich]

>>> On 28.06.13 at 18:10, George Dunlap <george.dunlap@eu.citrix.com> wrote:
> Either way, if someone *is* passing through a device, it is probably a 
> security issue.

After looking around a bit, I'm convinced that the other day I must
have tried this with xm (for whatever reason) - libxl has at least
two issues here compared to xend/xm:

- Missing error handling: Errors for the domain creation case
  propagate all the way back up to domcreate_attach_pci(), at
  which point they get silently ignored. Without having looked
  deeper, my first suggestion would be to simply drop devices
  from the d_config->pcidevs[] and decrement
  d_config->num_pcidevs upon failure, thus skipping any
  respective backend setup.

- Not making use of xc_test_assign_device() (which would allow
  detecting the situation before _any_ other setup gets done for
  the device).

Of course it is all but helpful that the backend in qemu has no way
of verifying that the device is in fact owned by the guest, i.e. it
has to blindly set up things trusting that the information handed
to it is consistent.

Jan

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[George Dunlap]

On Mon, Jul 1, 2013 at 11:53 AM, George Dunlap
<george.dunlap@eu.citrix.com> wrote:
> On 28/06/13 17:00, Jan Beulich wrote:
>>>>>
>>>>> On 28.06.13 at 17:37, George Dunlap <George.Dunlap@eu.citrix.com>
>>>>> wrote:
>>>
>>> - For HVM guests, the only user-visible indication tha the IOMMU has
>>> been disabled is the following error message on the command-line:
>>>
>>> # xl pci-attach h0 07:00.0
>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>>
>>> However, the device itself ends up passed-through to the guest anyway;
>>> the guest seems to be able to see it and interact with it normally.
>>> This is particularly scary, as in theory this should not be possible
>>> without a working IOMMU.
>>>
>>> I don't think this is a blocker for 4.3, but we should definitely
>>> release note it, and for 4.4 add a check to see if there is a
>>> functioning IOMMU and only add a device if there's an override set.
>>
>> To me this very much looks like a security problem (which I
>> think we should fix asap).
>
>
> Is it worth delaying the release (yet) another week for?
>
> Probably the simplest solution at the moment, if there's an easy way for the
> toolstack to figure out whether there is a working IOMMU or not, is to
> simply not allow pass-through without an IOMMU unless there is an override
> option.

On further reflection, I think there isn't actually a security bug
here: The promised behavior as of now is that if you really need to
have an iommu, then you should specify "iommu=force".  If I specify
iommu=force, then of course Xen doesn't boot, and I can't trigger this
problem.

This is actually a pretty awful interface, and should change, but
that's a 4.4 thing, not a 4.3 thing.  Since we haven't had any other
issues reported, I think we should go ahead with the scheduled release
tomorrow.

 -George

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Jan Beulich]

>>> On 01.07.13 at 12:53, George Dunlap <george.dunlap@eu.citrix.com> wrote:
> On 28/06/13 17:00, Jan Beulich wrote:
>>>>> On 28.06.13 at 17:37, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>>> - For HVM guests, the only user-visible indication tha the IOMMU has
>>> been disabled is the following error message on the command-line:
>>>
>>> # xl pci-attach h0 07:00.0
>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>>
>>> However, the device itself ends up passed-through to the guest anyway;
>>> the guest seems to be able to see it and interact with it normally.
>>> This is particularly scary, as in theory this should not be possible
>>> without a working IOMMU.
>>>
>>> I don't think this is a blocker for 4.3, but we should definitely
>>> release note it, and for 4.4 add a check to see if there is a
>>> functioning IOMMU and only add a device if there's an override set.
>> To me this very much looks like a security problem (which I
>> think we should fix asap).
> 
> Is it worth delaying the release (yet) another week for?

I would say so, but I'm open to being convinced otherwise.

> Probably the simplest solution at the moment, if there's an easy way for 
> the toolstack to figure out whether there is a working IOMMU or not, is 
> to simply not allow pass-through without an IOMMU unless there is an 
> override option.

xend had no override option - pass-through to HVM without
IOMMU should never be allowed imo.

Adding proper error handling perhaps is indeed beyond what's
reasonable for 4.3, but making use of xc_test_assign_device()
should result in not too big a change.

Jan

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Jan Beulich]

>>> On 01.07.13 at 14:15, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> On Mon, Jul 1, 2013 at 11:53 AM, George Dunlap
> <george.dunlap@eu.citrix.com> wrote:
>> On 28/06/13 17:00, Jan Beulich wrote:
>>>>>>
>>>>>> On 28.06.13 at 17:37, George Dunlap <George.Dunlap@eu.citrix.com>
>>>>>> wrote:
>>>>
>>>> - For HVM guests, the only user-visible indication tha the IOMMU has
>>>> been disabled is the following error message on the command-line:
>>>>
>>>> # xl pci-attach h0 07:00.0
>>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>>>
>>>> However, the device itself ends up passed-through to the guest anyway;
>>>> the guest seems to be able to see it and interact with it normally.
>>>> This is particularly scary, as in theory this should not be possible
>>>> without a working IOMMU.
>>>>
>>>> I don't think this is a blocker for 4.3, but we should definitely
>>>> release note it, and for 4.4 add a check to see if there is a
>>>> functioning IOMMU and only add a device if there's an override set.
>>>
>>> To me this very much looks like a security problem (which I
>>> think we should fix asap).
>>
>>
>> Is it worth delaying the release (yet) another week for?
>>
>> Probably the simplest solution at the moment, if there's an easy way for the
>> toolstack to figure out whether there is a working IOMMU or not, is to
>> simply not allow pass-through without an IOMMU unless there is an override
>> option.
> 
> On further reflection, I think there isn't actually a security bug
> here: The promised behavior as of now is that if you really need to
> have an iommu, then you should specify "iommu=force".  If I specify
> iommu=force, then of course Xen doesn't boot, and I can't trigger this
> problem.

I disagree, not the least because the behavior was different with
xend: When there's no IOMMU, pass-through to HVM must not
happen (or we'd have to suppress bus mastering on any such
passed through device). Pass-through to PV may happen, but is
insecure (as would be pass-through to HVM with disabled bus
mastering). So to anyone migrating from xend, if we don't change
things, this will at least be perceived as a security bug.

> This is actually a pretty awful interface, and should change, but
> that's a 4.4 thing, not a 4.3 thing.  Since we haven't had any other
> issues reported, I think we should go ahead with the scheduled release
> tomorrow.

As per the above and the earlier reply I sent, I don't think we
should release without this fixed. Let me see whether the minimal
fix I sketched out earlier works...

Jan

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Ian Jackson]

Jan Beulich writes ("Re: [Xen-devel] [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not"):
> As per the above and the earlier reply I sent, I don't think we
> should release without this fixed. Let me see whether the minimal
> fix I sketched out earlier works...

As I just wrote on IRC (Jan, you seem to have fallen off the channel);

14:08 <Diziet> Re Jan's comments.  To be clear, this isn't a regression 
               compared to 4.2, right ?
14:10 <Diziet> I think the right answer is to treat this as an embargo-less 
               security problem.  Preparing a proper security patch for that 
               probably involves fixing the error handling.
14:10 <Diziet> And that might take a week.
14:10 <Diziet> I'm really not keen on this "extra check" approach.
14:11 <Diziet> So I would go ahead with the release tomorrow.

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Jan Beulich]

>>> On 01.07.13 at 15:17, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Jan Beulich writes ("Re: [Xen-devel] [TESTDAY] PV / HVM pass-through works when 
> IOMMU present; weird failures when not"):
>> As per the above and the earlier reply I sent, I don't think we
>> should release without this fixed. Let me see whether the minimal
>> fix I sketched out earlier works...
> 
> As I just wrote on IRC (Jan, you seem to have fallen off the channel);
> 
> 14:08 <Diziet> Re Jan's comments.  To be clear, this isn't a regression 
>                compared to 4.2, right ?
> 14:10 <Diziet> I think the right answer is to treat this as an embargo-less 
>                security problem.  Preparing a proper security patch for that 
>                probably involves fixing the error handling.
> 14:10 <Diziet> And that might take a week.
> 14:10 <Diziet> I'm really not keen on this "extra check" approach.

What's wrong with it? It allows you to bail early, thus avoiding any
other setup to be done. But yes, it doesn't save you from (sooner
or later) doing proper error recovery.

Jan

> 14:11 <Diziet> So I would go ahead with the release tomorrow.

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Stefano Stabellini]

On Mon, 1 Jul 2013, Jan Beulich wrote:
> >>> On 01.07.13 at 14:15, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> > On Mon, Jul 1, 2013 at 11:53 AM, George Dunlap
> > <george.dunlap@eu.citrix.com> wrote:
> >> On 28/06/13 17:00, Jan Beulich wrote:
> >>>>>>
> >>>>>> On 28.06.13 at 17:37, George Dunlap <George.Dunlap@eu.citrix.com>
> >>>>>> wrote:
> >>>>
> >>>> - For HVM guests, the only user-visible indication tha the IOMMU has
> >>>> been disabled is the following error message on the command-line:
> >>>>
> >>>> # xl pci-attach h0 07:00.0
> >>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
> >>>>
> >>>> However, the device itself ends up passed-through to the guest anyway;
> >>>> the guest seems to be able to see it and interact with it normally.
> >>>> This is particularly scary, as in theory this should not be possible
> >>>> without a working IOMMU.
> >>>>
> >>>> I don't think this is a blocker for 4.3, but we should definitely
> >>>> release note it, and for 4.4 add a check to see if there is a
> >>>> functioning IOMMU and only add a device if there's an override set.
> >>>
> >>> To me this very much looks like a security problem (which I
> >>> think we should fix asap).
> >>
> >>
> >> Is it worth delaying the release (yet) another week for?
> >>
> >> Probably the simplest solution at the moment, if there's an easy way for the
> >> toolstack to figure out whether there is a working IOMMU or not, is to
> >> simply not allow pass-through without an IOMMU unless there is an override
> >> option.
> > 
> > On further reflection, I think there isn't actually a security bug
> > here: The promised behavior as of now is that if you really need to
> > have an iommu, then you should specify "iommu=force".  If I specify
> > iommu=force, then of course Xen doesn't boot, and I can't trigger this
> > problem.
> 
> I disagree, not the least because the behavior was different with
> xend: When there's no IOMMU, pass-through to HVM must not
> happen (or we'd have to suppress bus mastering on any such
> passed through device).
> Pass-through to PV may happen, but is
> insecure (as would be pass-through to HVM with disabled bus
> mastering).

I don't have an opinion on the release, but this is certainly true.

Re: [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

[Ian Jackson]

I have assigned this issue XSA-61.

Ian.