QEMU-Devel Archive on lore.kernel.org
 help / color / Atom feed
From: Jonathan Cameron <jonathan.cameron@huawei.com>
To: QEMU Developers <qemu-devel@nongnu.org>
Cc: Peter Maydell <peter.maydell@linaro.org>,
	qemu-arm <qemu-arm@nongnu.org>,
	linuxarm@huawei.com, jcm@redhat.com,
	Auger Eric <eric.auger@redhat.com>
Subject: Re: [Qemu-devel] [RFC PATCH 0/7] qemu: CCIX pcie config space emulation
Date: Tue, 6 Aug 2019 12:19:37 +0100
Message-ID: <20190806121937.00000c4a@huawei.com> (raw)
In-Reply-To: <20190625112752.83188-1-Jonathan.Cameron@huawei.com>

For reference alongside this patch set.

Evaluation version of the CCIX 1.0a base specification now available,
(though there is a form to complete and license agreement)..

https://www.ccixconsortium.com/ccix-library/download-form/

Thanks,

Jonathan

On Tue, 25 Jun 2019 19:27:45 +0800
Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:

> CCIX topologies are 'layered' on top of PCIe tree topologies.
> This is done primarily by allowing a single CCIX device to appear as
> multiple disjoint nodes in the PCIe tree.
> 
> This layering is described via extensive PCIe DVSEC extended
> capabilities in PCIe config space across all the functions that
> are present in the device (some placement rules apply).
> 
> The extremely flexible nature of allowed topologies makes the
> development of generic firmware and OS software difficult if we rely
> on actual hardware setups, due to the large test set that is necessary.
> 
> To enable the ongoing work on EDK2 and within the Linux kernel and
> userspace, this series provides the bare bones of what is necessary
> to be able to test 'configuration' of a CCIX topology.  Note
> that no actual CCIX data flow is being emulated within this patchset,
> merely a substantial subset of the interface that allows the topologies
> to be configured.  Testing has to rely on verification based on
> the result rather than true emulation of the coherency protocol.
> (that is a very different form of emulation for which other tools
> are perhaps better suited).
> 
> For information on how to do the coherency protocol configuration,
> see the forthcoming CCIX SW guide, in conjunction with the CCIX 1.0
> Base Specification.
> 
> An example of a 2x2 mesh with a spur to the host can be run with:
> 
> qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \
> ...
>  -device ioh3420,id=root_port1 \
>  -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev1",bus=root_port1,addr=0.0,multifunction="on",id=up0,port_id=0 \
>  -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=0,chassis=2,id=bus_top,port_id=1 \
>  -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=1,chassis=2,id=bus_left,port_id=2 \
>  -device ccix-ep,primaryport=false,home_agents=1,request_agents=1,ccix_device="ccix_dev1",bus=root_port1,addr=0.1,multifunction="on" \
>  -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev2",bus=bus_top,addr=0.0,multifunction="on",id=up1,port_id=0 \
>  -device ccix-downstream-port,ccix_device="ccix_dev2",bus=up1,slot=0,chassis=3,id=bus_right,port_id=1 \
>  -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev2",bus=bus_top,addr=0.1,multifunction="on" \
>  -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev3",bus=bus_left,addr=0.0,multifunction="on",id=up2,port_id=0 \
>  -device ccix-downstream-port,ccix_device="ccix_dev3",bus=up2,slot=0,chassis=4,id=bus_bottom,port_id=1,multifunciton="on" \
>  -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev3",bus=bus_left,addr=0.1,multifunction="on" \
>  -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_right,addr=0.0,port_id=0 \
>  -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_bottom,addr=0.0,port_id=1
> 
> 
> I'm not going to try drawing all the detail (it's bad enough
> trying to draw these in inkscape, but in a very much simplifed
> fashion, this generates.
> 
> -----------------
> |     Host      |
> |               | 
> --- root_port1--
>         |
>         |
>         v
> -----------------          ---------------
> |  ccix_dev1    | -------> |  ccix_dev2  |
> -----------------          ---------------
>         |                         |
>         V                         V
> -----------------          ---------------
> |  ccix_dev3    | -------> |  ccix_dev4  |
> -----------------          ---------------
> 
> $lspci -t
> -[0000:00]-+-00.0
>            +-01.0-[01-08]--+-00.0-[02-08]--+-00.0-[03-05]--+-00.0-[04-05]----00.0-[05]----00.0
>                            |               |               |               \-00.1
>                            |               |               \-01.0-[06-08]--+-00.0-[07-08]----00.0-[08]----00.0
>                            |               |                               \-00.1
>                            |               \-00.1
> 						       
> RFC questions:
> 
> 1.  The nature of CCIX devices is that we need to extend normal
>     PCI devices, slots, and ports.  I could not find an elegant way of
>     doing this without lots of code replication.  The current solution
>     just exposes some internal functions from xio3130 port implementations.
>     Is there a better way to do this?
> 
> 2.  The association of the different PCIDevices to a given CCIX device is
>     currently somewhat of a hack. Can anyone suggest a cleaner solution
>     for this?  I can improved the current implementation, but don't really
>     like that we basically search for all the parts whenever a cross
>     device implementation is needed.
> 
> 3.  Is emulation via a large number of PCIe devices like this a good
>     approach or is there a nicer way to handle it?  Given we need to
>     be able to 'spread' the CCIX device configuration across multiple
>     PCIe functions anyway perhaps this is the most sensible approach.
> 
> 4.  I've cut and paste a 100+ lines of code from each of the xio3130_*
>     modules as we are also implemening PCIE up and downstream ports
>     and as this is a emulation only device, we might as well use the
>     same register set.  There are various possible ways to avoid this:
>     * Add a library with the shared code in it.
>     * Add an xio3130_upstream.h header and similar to allow the CCIX
>       port modules to call those functions directly.
>     * Don't worry about the replication in the interests of keeping
>       the code structure simple.
> 
> 5.  I'm not that familiar with qemu 'style' yet, so pointers on structures
>     to use etc most welcome.
> 
> Note that not all elements of CCIX are supported by the current implementation,
> for example slave agents and error records are missing.  These will follow
> either in later revisions or as follow patches.  We also have no actual
> accelerator functions in the current design and hence no mapping to RAs.
> 
> Only a subset of configuration constraints are currently implemented.
> This will want tightenning up in future.
> 
> As we don't have any actual chunks of the spec in here so I haven't
> added the statement from the trademark grant that follows.
> 
> Thanks,
> 
> Jonathan
> 
> This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to
> you and other parties that are paticipating (the "participants") in
> qemu with the understanding that the participants will use CCIX's
> name and trademark only when this patch is used in association with
> qemu.
> 
> CCIX is also distributing this patch to these participants with the
> understanding that if any portion of the CCIX specification will be
> used or referenced in qemu, the participants will not modify the cited
> portion of the CCIX specification and will give CCIX propery copyright
> attribution by including the following copyright notice with
> the cited part of the CCIX specification:
> "© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED."
> 
> Jonathan Cameron (7):
>   Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
>   pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs.
>   pci: CCIX config space emulation library.
>   pci-bridge: CCIX capable PCIE/CCIX switch upstream port.
>   pci-bridge: CCIX capable PCIE/CCIX switch downstream port
>   misc: CCIX endpoint function
>   Temp: Add to ARM64 makefiles for testing
> 
>  default-configs/arm-softmmu.mak           |    3 +-
>  hw/misc/Kconfig                           |    5 +
>  hw/misc/Makefile.objs                     |    1 +
>  hw/misc/ccix-ep.c                         |  112 ++
>  hw/pci-bridge/Kconfig                     |    5 +
>  hw/pci-bridge/Makefile.objs               |    1 +
>  hw/pci-bridge/ccix_downstream.c           |  222 ++++
>  hw/pci-bridge/ccix_upstream.c             |  197 ++++
>  hw/pci/Kconfig                            |    3 +
>  hw/pci/Makefile.objs                      |    1 +
>  hw/pci/ccix_lib.c                         | 1299 +++++++++++++++++++++
>  include/hw/misc/ccix.h                    |   28 +
>  include/hw/pci/pci_ids.h                  |    7 +
>  include/standard-headers/linux/pci_regs.h |    3 +-
>  14 files changed, 1885 insertions(+), 2 deletions(-)
>  create mode 100644 hw/misc/ccix-ep.c
>  create mode 100644 hw/pci-bridge/ccix_downstream.c
>  create mode 100644 hw/pci-bridge/ccix_upstream.c
>  create mode 100644 hw/pci/ccix_lib.c
>  create mode 100644 include/hw/misc/ccix.h
> 




  parent reply index

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-25 11:27 Jonathan Cameron
2019-06-25 11:27 ` [Qemu-devel] [RFC PATCH 1/7] Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy Jonathan Cameron
2019-06-25 11:27 ` [Qemu-devel] [RFC PATCH 2/7] pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs Jonathan Cameron
2019-06-25 11:27 ` [Qemu-devel] [RFC PATCH 3/7] pci: CCIX config space emulation library Jonathan Cameron
2019-06-25 11:27 ` [Qemu-devel] [RFC PATCH 4/7] pci-bridge: CCIX capable PCIE/CCIX switch upstream port Jonathan Cameron
2019-06-25 11:27 ` [Qemu-devel] [RFC PATCH 5/7] pci-bridge: CCIX capable PCIE/CCIX switch downstream port Jonathan Cameron
2019-06-25 11:27 ` [Qemu-devel] [RFC PATCH 6/7] misc: CCIX endpoint function Jonathan Cameron
2019-06-25 11:27 ` [Qemu-devel] [RFC PATCH 7/7] Temp: Add to ARM64 makefiles for testing Jonathan Cameron
2019-08-06 11:19 ` Jonathan Cameron [this message]
2019-08-16 12:59 ` [Qemu-devel] [RFC PATCH 0/7] qemu: CCIX pcie config space emulation Peter Maydell
2019-08-19  9:47   ` Jonathan Cameron

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190806121937.00000c4a@huawei.com \
    --to=jonathan.cameron@huawei.com \
    --cc=eric.auger@redhat.com \
    --cc=jcm@redhat.com \
    --cc=linuxarm@huawei.com \
    --cc=peter.maydell@linaro.org \
    --cc=qemu-arm@nongnu.org \
    --cc=qemu-devel@nongnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

QEMU-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/qemu-devel/0 qemu-devel/git/0.git
	git clone --mirror https://lore.kernel.org/qemu-devel/1 qemu-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 qemu-devel qemu-devel/ https://lore.kernel.org/qemu-devel \
		qemu-devel@nongnu.org
	public-inbox-index qemu-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.nongnu.qemu-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git