Community call: PCI Emulation - Future Direction (Wed, May 2nd, UTC 16:00-17:00 / BST 17:00-18:00)

Community call: PCI Emulation - Future Direction (Wed, May 2nd, UTC 16:00-17:00 / BST 17:00-18:00)

[Lars Kurth]

Hi all,
as agreed please find attached the meeting invite
Regards
Lars

## Agenda (provisional)
I copied what was discussed on this thread so far https://docs.google.com/document/d/1RWylmNmBXOrgGLARj6_ynK50P7SZPl4LpnmhGaPglJw/edit?usp=sharing, which I will use as pad to write down minutes. Feel free to make agenda suggestions and copy relevant information into the doc, prior to the meeting.

## Time
Wed, May 2nd, UTC 16:00-17:00 
Wed, May 2nd, BST 17:00-18:00
For other times, see
https://www.timeanddate.com/worldclock/meetingdetails.html?year=2018&month=5&day=2&hour=16&min=0&sec=0&p1=136&p2=37&p3=224&p4=179&p5=240&p6=33

## Call details
https://www.gotomeet.me/LarsKurth 
First GoToMeeting? Let's do a quick system check: https://link.gotomeeting.com/system-check

You can also dial in using your phone. 

Access Code: 906-886-965

United Kingdom: +44 330 221 0088 
United States: +1 (571) 317-3129
Australia: +61 2 9087 3604
China (Toll Free): 4008 811084

More phone numbers 
Argentina (Toll Free): 0 800 444 3375 
Austria: +43 7 2081 5427 
Bahrain (Toll Free): 800 81 111 
Belarus (Toll Free): 8 820 0011 0400 
Belgium: +32 28 93 7018 
Brazil (Toll Free): 0 800 047 4906 
Bulgaria (Toll Free): 00800 120 4417 
Canada: +1 (647) 497-9391 
Chile (Toll Free): 800 395 150 
Colombia (Toll Free): 01 800 518 4483 
Czech Republic (Toll Free): 800 500448 
Denmark: +45 32 72 03 82 
Finland: +358 923 17 0568 
France: +33 170 950 594 
Germany: +49 692 5736 7317 
Greece (Toll Free): 00 800 4414 3838 
Hong Kong (Toll Free): 30713169 
Hungary (Toll Free): (06) 80 986 255 
Iceland (Toll Free): 800 7204 
India (Toll Free): 18002669272 
Indonesia (Toll Free): 007 803 020 5375 
Ireland: +353 15 360 728 
Israel (Toll Free): 1 809 454 830 
Italy: +39 0 247 92 13 01 
Japan (Toll Free): 0 120 663 800 
Korea, Republic of (Toll Free): 00798 14 207 4914 
Luxembourg (Toll Free): 800 85158 
Malaysia (Toll Free): 1 800 81 6854 
Mexico (Toll Free): 01 800 522 1133 
Netherlands: +31 207 941 377 
New Zealand: +64 9 280 6302 
Norway: +47 21 93 37 51 
Panama (Toll Free): 00 800 226 7928 
Peru (Toll Free): 0 800 77023 
Philippines (Toll Free): 1 800 1110 1661 
Poland (Toll Free): 00 800 1124759 
Portugal (Toll Free): 800 819 575 
Romania (Toll Free): 0 800 410 029 
Russian Federation (Toll Free): 8 800 100 6203 
Saudi Arabia (Toll Free): 800 844 3633 
Singapore (Toll Free): 18007231323 
South Africa (Toll Free): 0 800 555 447 
Spain: +34 932 75 2004 
Sweden: +46 853 527 827 
Switzerland: +41 225 4599 78 
Taiwan (Toll Free): 0 800 666 854 
Thailand (Toll Free): 001 800 011 023 
Turkey (Toll Free): 00 800 4488 23683 
Ukraine (Toll Free): 0 800 50 1733 
United Arab Emirates (Toll Free): 800 044 40439 
Uruguay (Toll Free): 0004 019 1018 
Viet Nam (Toll Free): 122 80 481 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Notes for upcoming PCI emulation call

[Alexey G]

I’ll try to summarize current issues/difficulties in extending the PCIe
passthrough support and possible ways to resolve these problems which
were discussed in the mailing list so far.

Possible options to extend PCI passthrough/emulation capabilities
-----------------------------------------------------------------

There is an arising need to support PCIe-specific features for PCI
passthrough. A lot of devices have PCIe Extended Capabilities above
100h offset. Even if we don’t want to support these capabilities in Xen
right away, a proprietary driver for a passed through device might want
to use these extended capabilities anyway -– Vendor-specific Extended
Capability is a classic example, though the device driver may try to
read other Extended Capabilities from its device’s conf space.

Apart from supporting PCIe Extended Capabilities, another possible (and
big) direction –- supporting PCIe-specific features in general like
native PCIe hotplug, new PM facilities or forwarding AER events to a
guest OS. This will require adding support for some cooperation between
passed through and emulated devices in a PCIe hierarchy, for which major
changes in emulated PCI bus architecture are needed. At the moment, all
PCIe devices are passed through in legacy PCI mode in Xen. This means
there is no support currently for PCIe-specific features like extended
PCI config space via ECAM.

Even providing support for PCIe Extended Capabilities alone requires
some changes –- we need to
1. Emulate ECAM (MMIO-accesses to MMCONFIG area) to allow
   reading/writing PCIe extended configuration space
2. Present a PCIe-capable system for a guest OS.

This can be achieved by adding QEMU Q35 emulation support to Xen (RFC
patch series for this feature was sent). For ECAM, in a very simplest
case, QEMU existing MMCONFIG emulation can be reused. However, there
are at least two incompatibility problems which need solution. These
are:

- Multiple PCI device emulators feature, used by VGPU in XenServer

- Emulating (a simplest) upstream PCIe hierarchy for passed through PCIe
devices. The issue was described in details here:
http://lists.gnu.org/archive/html/qemu-devel/2018-03/msg03593.html

Latter problem must be resolved properly by introducing emulated PCIe
Root Ports for passed through devices. Basically this means we need to
emulate PCI-PCI bridges with secondary bus used to place real passed
through devices, ideally using function grouping for related devices
like GPU and its HDAudio function.

There are different approaches _who_ should emulate these PCI-PCI
bridges. QEMU has support for emulated RPs and PCIe switches but we
might want to remove that privilege from QEMU as emulating RPs/switches
above _real_ passed through PCIe devices is a relatively system thing.
Also, we need to consider future PCIe passthru extensions like handling
PM events from passed through PCIe devices as these features assume some
additional support in upstream PCIe hierarchy.

So, we need to decide who will be controlling emulated Root Ports for
passed through devices – either Xen or QEMU. For a number of reasons it
will be beneficial to do it on Xen side while sticking to QEMU allows
reusing existing functionality on the other hand.

Now, regarding the multiple PCI device emulators. For multiple PCI
device emulators a specific passed through device may be assigned to a
separate device model (non-QEMU). At the low-level this will appear as
more than one IOREQ server present –- most PCI devices will be still
handled by QEMU, with some being assigned to another (device-specific)
device model -– a distinct binary –- via same
xc_hvm_map_pcidev_to_ioreq_server() call. Later,
hvm_select_ioreq_server() will select a proper device model destination
based on BDF location of the device and ioreqs will be sent to the
chosen target.
This works well for legacy CF8h/CFCh PCI conf accesses, but MMCONFIG
support introduces some problems.

First of all, MMCONFIG itself is a chipset-specific thing. Both
registers which control it and the number of MMCONFIG ranges
(ECAM-capable PCIe segments) may differ for different emulated
machines. This means that some designated device model should control
it according to the user-selected emulated machine. Device-specific
device model doesn't know anything about the emulated machine.

Secondly, in order to have all necessary information to forward ioreqs
to the correct device model, Xen needs to know
1. MMCONFIG base address and size (ideally extendable to support
   multiple MMCONFIGs)
2. MMCONFIG layout, corresponding to the current map of the PCI bus.
   This layout may change anytime due to a PCI-PCI bridge
   re-initialization or hotplugging a device.

There are different options how to pass this information to Xen. Xen
may even control it itself in some solutions.

MMCONFIG layout can be obtained passively, by simply observing
map_pcidev_to_ioreq_server calls to determine and store all emulated
PCI device BDF locations.

Another thing to consider here is MMIO hole layout and its impact.
For example, adding PCI-PCI bridges creates some complication as they
will provide windows in IO/MMIO space which should be sized accordingly
to the secondary PCI bus content. In some cases like hotplugging a PCIe
device (which should belong to some RP or switch DP) existing bridge
windows might be too small to provide space for a newly added device,
triggering PCI-PCI bridge and BARs re-initialization (aka PCI resource
rebalancing in Windows terms) in guest. This action may change the PCI
bus layout which needs to be addressed somehow. Also, by utilizing ACPI
_DSM method (not our case luckily as we don't provide it) Windows may
invoke a complete PCI BARs/PCI-PCI bridge re-initialization
unconditionally on system boot.


Possible directions to make multiple PCI device emulators compatible
with PCIe/MMCONFIG
--------------------------------------------------------------------

I. “Notification” approach. In this case QEMU will continue to emulate
PCIEXBAR and handle MMCONFIG accesses. But, upon encountering any
changes in the PCIEXBAR value, QEMU will report this change to Xen via
any suitable channel -– either a dedicated dmop, XenStore param or
anything else. Xen will store this information and use it to select a
proper IOREQ server destination for trapped MMCONFIG accesses.

II. “Own chipset device model”. In this case Xen will emulate some
chipset-specific devices himself. Of particular interest are MCH and
ICH9. Both emulated Root Complex and Root Ports will belong to Xen,
allowing implementing PCIe-specific features like AER reporting in any
convenient way. Ideally, from QEMU side only a set of distinct
PCIDevice’s will remain – storage, networking, etc. A dummy pci-host
will be providing forwarding of IOREQ_TYPE_PCI_CONFIG-accesses for
remaining PCIDevices. PCI bus layout seen by QEMU can be made different
with the real layout seen by guest. Final result will look like a new,
very reduced QEMU machine with dummy PCIBus/ISABus, perhaps even based
on top of QEMU null machine.

While this approach is beneficial in many ways, it will affect
compatibility with QEMU very, very badly. For example, NVDIMM support
patches from Intel rely on QEMU ACPI facilities which can become
completely inoperational due to removing emulated NB+SB and their
corresponding subtypes and properties. Multiple similar issues and
breakages may arise in future, though QEMU PM/ACPI facilities is the
main problem. Note that Xen already emulates some of PMBASE registers
and PMBASE value itself is hardcoded (at B000h IIRC). Own PMBASE
BAR emulation will allow to remove this limitation.

III. “Transparent emulation”. In this case Xen will intercept only some
known registers for chipset-specific devices emulated by QEMU.
PCIEXBAR, PMBASE, possibly MMIO Hole-controlling registers and some
others. A handler for this kind of registers can be selectively called
before or after the corresponding DM emulation (on different stages of
IOREQ processing) and should have freedom to specify whether the DM may
see this read/write (otherwise it is handled internally). This will
allow to provide own support for PCIEXBAR/MMCONFIG emulation while
keeping compatibility with QEMU. Zero changes will be needed on QEMU
side.
Xen will detect the emulated chipset either passively or via sending
IOREQ_TYPE_PCI_CONFIG to read VID/DID from the device model directly.
NB/SB VID/DID values will be used to distinguish between different
emulated machines and to setup correct handlers for chipset-specific
registers. 

Due to the requirement for a PCIe device to cooperate with upstream
PCIe hierarchy (at least to belong to some RP/switch), some changes for
multiple PCI emulator support must be made no matter the chosen
solution.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Notes for upcoming PCI emulation call

[Roger Pau Monné]

On Wed, May 02, 2018 at 08:03:52PM +1000, Alexey G wrote:
> - Emulating (a simplest) upstream PCIe hierarchy for passed through PCIe
> devices. The issue was described in details here:
> http://lists.gnu.org/archive/html/qemu-devel/2018-03/msg03593.html
> 
> Latter problem must be resolved properly by introducing emulated PCIe
> Root Ports for passed through devices.

In order to solve it properly bridges are required. But as you point
out in your email, it can be worked around. Since there's already
quite a lot of stuff to discuss, and AFAICT bridge support is not
mandatory for an initial implementation I would suggest that the
bridge discussion is deferred until MCFG support for bus 0 is
implemented.

> While this approach is beneficial in many ways, it will affect
> compatibility with QEMU very, very badly. For example, NVDIMM support
> patches from Intel rely on QEMU ACPI facilities which can become
> completely inoperational due to removing emulated NB+SB and their
> corresponding subtypes and properties.

There are multiple issues with NVDIMM on Xen, and we are still
discussing how NVDIMM devices should be handled on Dom0, I think at
this stage ACPI is a quite minor issue compared to the other
challenges presented by NVDIMM.

And in any case in order for vNVDIMM to support PVH guests the QEMU
code is certainly not going to be used on Xen.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Community call: PCI Emulation - Future Direction (Wed, May 2nd, UTC 16:00-17:00 / BST 17:00-18:00)

[Rich Persaud]

[-- Attachment #1.1: Type: text/plain, Size: 2192 bytes --]

> On Apr 24, 2018, at 05:19, Lars Kurth <lars.kurth@citrix.com> wrote:
> 
> Hi all,
> as agreed please find attached the meeting invite
> Regards
> Lars
> 
> ## Agenda (provisional)
> I copied what was discussed on this thread so far https://docs.google.com/document/d/1RWylmNmBXOrgGLARj6_ynK50P7SZPl4LpnmhGaPglJw/edit?usp=sharing, which I will use as pad to write down minutes. Feel free to make agenda suggestions and copy relevant information into the doc, prior to the meeting.

I would like to add an agenda item to discuss the level of security support that will be asserted in SUPPORT.md for driver domains which contain untrusted PCI devices.  Will Xen security support be different for SR-IOV devices?  GPUs vs. NICs?

There have been past discussions on this topic and a proposed PCI-iommu-bugs.txt file to help Xen users and developers understand the risks [2][3][4] that may arise from a hostile device and potentially buggy firmware.  If we can document specific risks, we can ask firmware developers to make specific improvements to improve the security of PCI emulation.

There is an active effort [4] underway to improve firmware security in servers (and eventually desktops), including a reduction of attack surface due to SMM.  There is also work underway [5][6] to perform secure boot between individual PCI devices and server motherboards.  Some of these concepts may already be deployed in Azure.

Several stakeholders will be attending or presenting at the PSEC [6] conference.

Rich

[1] Performance Isolation Exposure in Virtualized Platforms with PCI Passthrough I/O Sharing, https://mediatum.ub.tum.de/doc/1187609/972322.pdf

[2]  Securing Self-Virtualizing Ethernet Devices, https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-smolyar.pdf

[3]  Denial-of-Service Attacks on PCI Passthrough Devices, http://publications.andre-richter.com/richter2015denial.pdf

[4] Open Compute Open System Firmware, http://www.opencompute.org/wiki/Open_System_Firmware

[5] Open Compute Security, http://www.opencompute.org/wiki/Security

[6] Firmware attestation: https://www.platformsecuritysummit.com/prepare/#attestation


[-- Attachment #1.2: Type: text/html, Size: 3563 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel