SecureBoot and PCI passthrough with kernel lockdown in place (on Xen)

SecureBoot and PCI passthrough with kernel lockdown in place (on Xen)

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 2981 bytes --]

Hello,

We have run into an issue when trying to use PCI passthrough for a Xen
VM running on an host where dom0 kernel is 5.14.21 (but we think it
could be any kernel > 5.4) and SecureBoot is enabled.

The error we get, when (for instance) trying to attach a device to an
(HVM) VM, on such system is:

# xl pci-attach 2-fv-sles15sp4beta2 0000:58:03.0 
libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 12:Failed to initialize 12/15, type = 0x1, rc: -1
libxl: error: libxl_pci.c:1777:device_pci_add_done: Domain 12:libxl__device_pci_add failed for PCI device 0:58:3.0 (rc -28)
libxl: error: libxl_device.c:1420:device_addrm_aocomplete: unable to add device

QEMU, is telling us the following:

[00:04.0] xen_pt_msix_init: Error: Can't open /dev/mem: Operation not permitted
[00:04.0] xen_pt_msix_size_init: Error: Internal error: Invalid xen_pt_msix_init.

And the kernel reports this:

Jan 27 16:20:53 narvi-sr860v2-bps-sles15sp4b2 kernel: Lockdown: qemu-system-i38: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7

So, it's related to lockdown. Which AFAIUI it's consistent with the
fact that the problem only shows up when SecureBoot is enabled, as
that's implies lockdown. It's also consistent with the fact that we
don't seem to have any problems doing the same with a 5.3.x dom0
kernel... As there's no lockdown there!

Some digging revealed that QEMU tries to open /dev/mem in
xen_pt_msix_init():

    fd = open("/dev/mem", O_RDWR);
    ...
    msix->phys_iomem_base =
            mmap(NULL,
                 total_entries * PCI_MSIX_ENTRY_SIZE + msix->table_offset_adjust,
                 PROT_READ,
                 MAP_SHARED | MAP_LOCKED,
                 fd,
                 msix->table_base + table_off - msix->table_offset_adjust);
    close(fd);

This comes from commit:

commit 3854ca577dad92c4fe97b4a6ebce360e25407af7
Author: Jiang Yunhong <yunhong.jiang@intel.com>
Date:   Thu Jun 21 15:42:35 2012 +0000

    Introduce Xen PCI Passthrough, MSI
    
    A more complete history can be found here:
    git://xenbits.xensource.com/qemu-xen-unstable.git
    
    Signed-off-by: Jiang Yunhong <yunhong.jiang@intel.com>
    Signed-off-by: Shan Haitao <haitao.shan@intel.com>
    Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
    Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

Now, the questions:
- is this (i.e., PCI-Passthrough with a locked-down dom0 kernel) 
  working for anyone? I've Cc-ed Marek, because I think I've read that 
  QubesOS that it does on QubesOS, but I'm not sure if the situation 
  is the same...
- if it's working, how?

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: SecureBoot and PCI passthrough with kernel lockdown in place (on Xen)

[Jan Beulich]

On 14.02.2022 16:02, Dario Faggioli wrote:
> We have run into an issue when trying to use PCI passthrough for a Xen
> VM running on an host where dom0 kernel is 5.14.21 (but we think it
> could be any kernel > 5.4) and SecureBoot is enabled.
> 
> The error we get, when (for instance) trying to attach a device to an
> (HVM) VM, on such system is:
> 
> # xl pci-attach 2-fv-sles15sp4beta2 0000:58:03.0 
> libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 12:Failed to initialize 12/15, type = 0x1, rc: -1
> libxl: error: libxl_pci.c:1777:device_pci_add_done: Domain 12:libxl__device_pci_add failed for PCI device 0:58:3.0 (rc -28)
> libxl: error: libxl_device.c:1420:device_addrm_aocomplete: unable to add device
> 
> QEMU, is telling us the following:
> 
> [00:04.0] xen_pt_msix_init: Error: Can't open /dev/mem: Operation not permitted
> [00:04.0] xen_pt_msix_size_init: Error: Internal error: Invalid xen_pt_msix_init.
> 
> And the kernel reports this:
> 
> Jan 27 16:20:53 narvi-sr860v2-bps-sles15sp4b2 kernel: Lockdown: qemu-system-i38: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
> 
> So, it's related to lockdown. Which AFAIUI it's consistent with the
> fact that the problem only shows up when SecureBoot is enabled, as
> that's implies lockdown. It's also consistent with the fact that we
> don't seem to have any problems doing the same with a 5.3.x dom0
> kernel... As there's no lockdown there!
> 
> Some digging revealed that QEMU tries to open /dev/mem in
> xen_pt_msix_init():
> 
>     fd = open("/dev/mem", O_RDWR);
>     ...
>     msix->phys_iomem_base =
>             mmap(NULL,
>                  total_entries * PCI_MSIX_ENTRY_SIZE + msix->table_offset_adjust,
>                  PROT_READ,
>                  MAP_SHARED | MAP_LOCKED,
>                  fd,
>                  msix->table_base + table_off - msix->table_offset_adjust);
>     close(fd);

I think this is finally a clear indication that it has always been
wrong for qemu to access hardware directly like this. I see no way
around replacing this by something which isn't a bodge / layering
violation.

Jan

> This comes from commit:
> 
> commit 3854ca577dad92c4fe97b4a6ebce360e25407af7
> Author: Jiang Yunhong <yunhong.jiang@intel.com>
> Date:   Thu Jun 21 15:42:35 2012 +0000
> 
>     Introduce Xen PCI Passthrough, MSI
>     
>     A more complete history can be found here:
>     git://xenbits.xensource.com/qemu-xen-unstable.git
>     
>     Signed-off-by: Jiang Yunhong <yunhong.jiang@intel.com>
>     Signed-off-by: Shan Haitao <haitao.shan@intel.com>
>     Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
>     Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> 
> Now, the questions:
> - is this (i.e., PCI-Passthrough with a locked-down dom0 kernel) 
>   working for anyone? I've Cc-ed Marek, because I think I've read that 
>   QubesOS that it does on QubesOS, but I'm not sure if the situation 
>   is the same...
> - if it's working, how?
> 
> Thanks and Regards

Re: SecureBoot and PCI passthrough with kernel lockdown in place (on Xen)

[Andrew Cooper]

On 14/02/2022 15:02, Dario Faggioli wrote:
> Hello,
>
> We have run into an issue when trying to use PCI passthrough for a Xen
> VM running on an host where dom0 kernel is 5.14.21 (but we think it
> could be any kernel > 5.4) and SecureBoot is enabled.

Back up a bit...

Xen doesn't support SecureBoot and there's a massive pile of work to
make it function, let alone work in a way that MSFT aren't liable to
revoke your cert on 0 notice.

>
> The error we get, when (for instance) trying to attach a device to an
> (HVM) VM, on such system is:
>
> # xl pci-attach 2-fv-sles15sp4beta2 0000:58:03.0 
> libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 12:Failed to initialize 12/15, type = 0x1, rc: -1
> libxl: error: libxl_pci.c:1777:device_pci_add_done: Domain 12:libxl__device_pci_add failed for PCI device 0:58:3.0 (rc -28)
> libxl: error: libxl_device.c:1420:device_addrm_aocomplete: unable to add device
>
> QEMU, is telling us the following:
>
> [00:04.0] xen_pt_msix_init: Error: Can't open /dev/mem: Operation not permitted
> [00:04.0] xen_pt_msix_size_init: Error: Internal error: Invalid xen_pt_msix_init.
>
> And the kernel reports this:
>
> Jan 27 16:20:53 narvi-sr860v2-bps-sles15sp4b2 kernel: Lockdown: qemu-system-i38: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
>
> So, it's related to lockdown. Which AFAIUI it's consistent with the
> fact that the problem only shows up when SecureBoot is enabled, as
> that's implies lockdown. It's also consistent with the fact that we
> don't seem to have any problems doing the same with a 5.3.x dom0
> kernel... As there's no lockdown there!
>
> Some digging revealed that QEMU tries to open /dev/mem in
> xen_pt_msix_init():
>
>     fd = open("/dev/mem", O_RDWR);
>     ...
>     msix->phys_iomem_base =
>             mmap(NULL,
>                  total_entries * PCI_MSIX_ENTRY_SIZE + msix->table_offset_adjust,
>                  PROT_READ,
>                  MAP_SHARED | MAP_LOCKED,
>                  fd,
>                  msix->table_base + table_off - msix->table_offset_adjust);
>     close(fd);

Yes.  Use of /dev/mem is not permitted in lockdown mode.  This wants
reworking into something which is lockdown compatible.

The real elephant in the room is that privcmd is not remotely safe to
use in a SecureBoot environment, because it lets any root userspace
trivially escalate privilege into the dom0 kernel, bypassing the
specific protection that SecureBoot is trying to achieve.

~Andrew

Re: SecureBoot and PCI passthrough with kernel lockdown in place (on Xen)

[marmarek]

[-- Attachment #1: Type: text/plain, Size: 3094 bytes --]

On Mon, Feb 14, 2022 at 03:25:31PM +0000, Andrew Cooper wrote:
> On 14/02/2022 15:02, Dario Faggioli wrote:
> > Hello,
> >
> > We have run into an issue when trying to use PCI passthrough for a Xen
> > VM running on an host where dom0 kernel is 5.14.21 (but we think it
> > could be any kernel > 5.4) and SecureBoot is enabled.
> 
> Back up a bit...
> 
> Xen doesn't support SecureBoot and there's a massive pile of work to
> make it function, let alone work in a way that MSFT aren't liable to
> revoke your cert on 0 notice.
> 
> >
> > The error we get, when (for instance) trying to attach a device to an
> > (HVM) VM, on such system is:
> >
> > # xl pci-attach 2-fv-sles15sp4beta2 0000:58:03.0 
> > libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 12:Failed to initialize 12/15, type = 0x1, rc: -1
> > libxl: error: libxl_pci.c:1777:device_pci_add_done: Domain 12:libxl__device_pci_add failed for PCI device 0:58:3.0 (rc -28)
> > libxl: error: libxl_device.c:1420:device_addrm_aocomplete: unable to add device
> >
> > QEMU, is telling us the following:
> >
> > [00:04.0] xen_pt_msix_init: Error: Can't open /dev/mem: Operation not permitted
> > [00:04.0] xen_pt_msix_size_init: Error: Internal error: Invalid xen_pt_msix_init.
> >
> > And the kernel reports this:
> >
> > Jan 27 16:20:53 narvi-sr860v2-bps-sles15sp4b2 kernel: Lockdown: qemu-system-i38: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
> >
> > So, it's related to lockdown. Which AFAIUI it's consistent with the
> > fact that the problem only shows up when SecureBoot is enabled, as
> > that's implies lockdown. It's also consistent with the fact that we
> > don't seem to have any problems doing the same with a 5.3.x dom0
> > kernel... As there's no lockdown there!
> >
> > Some digging revealed that QEMU tries to open /dev/mem in
> > xen_pt_msix_init():
> >
> >     fd = open("/dev/mem", O_RDWR);
> >     ...
> >     msix->phys_iomem_base =
> >             mmap(NULL,
> >                  total_entries * PCI_MSIX_ENTRY_SIZE + msix->table_offset_adjust,
> >                  PROT_READ,
> >                  MAP_SHARED | MAP_LOCKED,
> >                  fd,
> >                  msix->table_base + table_off - msix->table_offset_adjust);
> >     close(fd);
> 
> Yes.  Use of /dev/mem is not permitted in lockdown mode.  This wants
> reworking into something which is lockdown compatible.

FWIW, Qubes has PCI passthrough working with qemu in stubdomain, which
works without access to /dev/mem in dom0. We do this, by disabling
MSI-X, including the above piece of code...

https://github.com/QubesOS/qubes-vmm-xen-stubdom-linux/blob/master/qemu/patches/0005-Disable-MSI-X-caps.patch

> The real elephant in the room is that privcmd is not remotely safe to
> use in a SecureBoot environment, because it lets any root userspace
> trivially escalate privilege into the dom0 kernel, bypassing the
> specific protection that SecureBoot is trying to achieve.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: SecureBoot and PCI passthrough with kernel lockdown in place (on Xen)

[marmarek]

[-- Attachment #1: Type: text/plain, Size: 3094 bytes --]

On Mon, Feb 14, 2022 at 03:25:31PM +0000, Andrew Cooper wrote:
> On 14/02/2022 15:02, Dario Faggioli wrote:
> > Hello,
> >
> > We have run into an issue when trying to use PCI passthrough for a Xen
> > VM running on an host where dom0 kernel is 5.14.21 (but we think it
> > could be any kernel > 5.4) and SecureBoot is enabled.
> 
> Back up a bit...
> 
> Xen doesn't support SecureBoot and there's a massive pile of work to
> make it function, let alone work in a way that MSFT aren't liable to
> revoke your cert on 0 notice.
> 
> >
> > The error we get, when (for instance) trying to attach a device to an
> > (HVM) VM, on such system is:
> >
> > # xl pci-attach 2-fv-sles15sp4beta2 0000:58:03.0 
> > libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 12:Failed to initialize 12/15, type = 0x1, rc: -1
> > libxl: error: libxl_pci.c:1777:device_pci_add_done: Domain 12:libxl__device_pci_add failed for PCI device 0:58:3.0 (rc -28)
> > libxl: error: libxl_device.c:1420:device_addrm_aocomplete: unable to add device
> >
> > QEMU, is telling us the following:
> >
> > [00:04.0] xen_pt_msix_init: Error: Can't open /dev/mem: Operation not permitted
> > [00:04.0] xen_pt_msix_size_init: Error: Internal error: Invalid xen_pt_msix_init.
> >
> > And the kernel reports this:
> >
> > Jan 27 16:20:53 narvi-sr860v2-bps-sles15sp4b2 kernel: Lockdown: qemu-system-i38: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
> >
> > So, it's related to lockdown. Which AFAIUI it's consistent with the
> > fact that the problem only shows up when SecureBoot is enabled, as
> > that's implies lockdown. It's also consistent with the fact that we
> > don't seem to have any problems doing the same with a 5.3.x dom0
> > kernel... As there's no lockdown there!
> >
> > Some digging revealed that QEMU tries to open /dev/mem in
> > xen_pt_msix_init():
> >
> >     fd = open("/dev/mem", O_RDWR);
> >     ...
> >     msix->phys_iomem_base =
> >             mmap(NULL,
> >                  total_entries * PCI_MSIX_ENTRY_SIZE + msix->table_offset_adjust,
> >                  PROT_READ,
> >                  MAP_SHARED | MAP_LOCKED,
> >                  fd,
> >                  msix->table_base + table_off - msix->table_offset_adjust);
> >     close(fd);
> 
> Yes.  Use of /dev/mem is not permitted in lockdown mode.  This wants
> reworking into something which is lockdown compatible.

FWIW, Qubes has PCI passthrough working with qemu in stubdomain, which
works without access to /dev/mem in dom0. We do this, by disabling
MSI-X, including the above piece of code...

https://github.com/QubesOS/qubes-vmm-xen-stubdom-linux/blob/master/qemu/patches/0005-Disable-MSI-X-caps.patch

> The real elephant in the room is that privcmd is not remotely safe to
> use in a SecureBoot environment, because it lets any root userspace
> trivially escalate privilege into the dom0 kernel, bypassing the
> specific protection that SecureBoot is trying to achieve.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]