Re: [Qemu-devel] [RFC PATCH 0/7] qemu: CCIX pcie config space emulation

[Jonathan Cameron <jonathan.cameron@huawei.com>]

For reference alongside this patch set.

Evaluation version of the CCIX 1.0a base specification now available,
(though there is a form to complete and license agreement)..

https://www.ccixconsortium.com/ccix-library/download-form/

Thanks,

Jonathan

On Tue, 25 Jun 2019 19:27:45 +0800
Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:

> CCIX topologies are 'layered' on top of PCIe tree topologies.
> This is done primarily by allowing a single CCIX device to appear as
> multiple disjoint nodes in the PCIe tree.
> 
> This layering is described via extensive PCIe DVSEC extended
> capabilities in PCIe config space across all the functions that
> are present in the device (some placement rules apply).
> 
> The extremely flexible nature of allowed topologies makes the
> development of generic firmware and OS software difficult if we rely
> on actual hardware setups, due to the large test set that is necessary.
> 
> To enable the ongoing work on EDK2 and within the Linux kernel and
> userspace, this series provides the bare bones of what is necessary
> to be able to test 'configuration' of a CCIX topology.  Note
> that no actual CCIX data flow is being emulated within this patchset,
> merely a substantial subset of the interface that allows the topologies
> to be configured.  Testing has to rely on verification based on
> the result rather than true emulation of the coherency protocol.
> (that is a very different form of emulation for which other tools
> are perhaps better suited).
> 
> For information on how to do the coherency protocol configuration,
> see the forthcoming CCIX SW guide, in conjunction with the CCIX 1.0
> Base Specification.
> 
> An example of a 2x2 mesh with a spur to the host can be run with:
> 
> qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \
> ...
>  -device ioh3420,id=root_port1 \
>  -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev1",bus=root_port1,addr=0.0,multifunction="on",id=up0,port_id=0 \
>  -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=0,chassis=2,id=bus_top,port_id=1 \
>  -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=1,chassis=2,id=bus_left,port_id=2 \
>  -device ccix-ep,primaryport=false,home_agents=1,request_agents=1,ccix_device="ccix_dev1",bus=root_port1,addr=0.1,multifunction="on" \
>  -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev2",bus=bus_top,addr=0.0,multifunction="on",id=up1,port_id=0 \
>  -device ccix-downstream-port,ccix_device="ccix_dev2",bus=up1,slot=0,chassis=3,id=bus_right,port_id=1 \
>  -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev2",bus=bus_top,addr=0.1,multifunction="on" \
>  -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev3",bus=bus_left,addr=0.0,multifunction="on",id=up2,port_id=0 \
>  -device ccix-downstream-port,ccix_device="ccix_dev3",bus=up2,slot=0,chassis=4,id=bus_bottom,port_id=1,multifunciton="on" \
>  -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev3",bus=bus_left,addr=0.1,multifunction="on" \
>  -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_right,addr=0.0,port_id=0 \
>  -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_bottom,addr=0.0,port_id=1
> 
> 
> I'm not going to try drawing all the detail (it's bad enough
> trying to draw these in inkscape, but in a very much simplifed
> fashion, this generates.
> 
> -----------------
> |     Host      |
> |               | 
> --- root_port1--
>         |
>         |
>         v
> -----------------          ---------------
> |  ccix_dev1    | -------> |  ccix_dev2  |
> -----------------          ---------------
>         |                         |
>         V                         V
> -----------------          ---------------
> |  ccix_dev3    | -------> |  ccix_dev4  |
> -----------------          ---------------
> 
> $lspci -t
> -[0000:00]-+-00.0
>            +-01.0-[01-08]--+-00.0-[02-08]--+-00.0-[03-05]--+-00.0-[04-05]----00.0-[05]----00.0
>                            |               |               |               \-00.1
>                            |               |               \-01.0-[06-08]--+-00.0-[07-08]----00.0-[08]----00.0
>                            |               |                               \-00.1
>                            |               \-00.1
> 						       
> RFC questions:
> 
> 1.  The nature of CCIX devices is that we need to extend normal
>     PCI devices, slots, and ports.  I could not find an elegant way of
>     doing this without lots of code replication.  The current solution
>     just exposes some internal functions from xio3130 port implementations.
>     Is there a better way to do this?
> 
> 2.  The association of the different PCIDevices to a given CCIX device is
>     currently somewhat of a hack. Can anyone suggest a cleaner solution
>     for this?  I can improved the current implementation, but don't really
>     like that we basically search for all the parts whenever a cross
>     device implementation is needed.
> 
> 3.  Is emulation via a large number of PCIe devices like this a good
>     approach or is there a nicer way to handle it?  Given we need to
>     be able to 'spread' the CCIX device configuration across multiple
>     PCIe functions anyway perhaps this is the most sensible approach.
> 
> 4.  I've cut and paste a 100+ lines of code from each of the xio3130_*
>     modules as we are also implemening PCIE up and downstream ports
>     and as this is a emulation only device, we might as well use the
>     same register set.  There are various possible ways to avoid this:
>     * Add a library with the shared code in it.
>     * Add an xio3130_upstream.h header and similar to allow the CCIX
>       port modules to call those functions directly.
>     * Don't worry about the replication in the interests of keeping
>       the code structure simple.
> 
> 5.  I'm not that familiar with qemu 'style' yet, so pointers on structures
>     to use etc most welcome.
> 
> Note that not all elements of CCIX are supported by the current implementation,
> for example slave agents and error records are missing.  These will follow
> either in later revisions or as follow patches.  We also have no actual
> accelerator functions in the current design and hence no mapping to RAs.
> 
> Only a subset of configuration constraints are currently implemented.
> This will want tightenning up in future.
> 
> As we don't have any actual chunks of the spec in here so I haven't
> added the statement from the trademark grant that follows.
> 
> Thanks,
> 
> Jonathan
> 
> This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to
> you and other parties that are paticipating (the "participants") in
> qemu with the understanding that the participants will use CCIX's
> name and trademark only when this patch is used in association with
> qemu.
> 
> CCIX is also distributing this patch to these participants with the
> understanding that if any portion of the CCIX specification will be
> used or referenced in qemu, the participants will not modify the cited
> portion of the CCIX specification and will give CCIX propery copyright
> attribution by including the following copyright notice with
> the cited part of the CCIX specification:
> "© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED."
> 
> Jonathan Cameron (7):
>   Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
>   pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs.
>   pci: CCIX config space emulation library.
>   pci-bridge: CCIX capable PCIE/CCIX switch upstream port.
>   pci-bridge: CCIX capable PCIE/CCIX switch downstream port
>   misc: CCIX endpoint function
>   Temp: Add to ARM64 makefiles for testing
> 
>  default-configs/arm-softmmu.mak           |    3 +-
>  hw/misc/Kconfig                           |    5 +
>  hw/misc/Makefile.objs                     |    1 +
>  hw/misc/ccix-ep.c                         |  112 ++
>  hw/pci-bridge/Kconfig                     |    5 +
>  hw/pci-bridge/Makefile.objs               |    1 +
>  hw/pci-bridge/ccix_downstream.c           |  222 ++++
>  hw/pci-bridge/ccix_upstream.c             |  197 ++++
>  hw/pci/Kconfig                            |    3 +
>  hw/pci/Makefile.objs                      |    1 +
>  hw/pci/ccix_lib.c                         | 1299 +++++++++++++++++++++
>  include/hw/misc/ccix.h                    |   28 +
>  include/hw/pci/pci_ids.h                  |    7 +
>  include/standard-headers/linux/pci_regs.h |    3 +-
>  14 files changed, 1885 insertions(+), 2 deletions(-)
>  create mode 100644 hw/misc/ccix-ep.c
>  create mode 100644 hw/pci-bridge/ccix_downstream.c
>  create mode 100644 hw/pci-bridge/ccix_upstream.c
>  create mode 100644 hw/pci/ccix_lib.c
>  create mode 100644 include/hw/misc/ccix.h
>