All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
@ 2018-06-28  8:36 Cédric Le Goater
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 1/2] " Cédric Le Goater
                   ` (2 more replies)
  0 siblings, 3 replies; 29+ messages in thread
From: Cédric Le Goater @ 2018-06-28  8:36 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin, Cédric Le Goater

This is a model of the PCIe Host Bridge (PHB3) controller found on a
Power8 processor. The Power8 processor comes in different flavors:
Venice, Murano, Naple, each having a different number of PHBs. Multi
chip is supported, each chip adding its set of PHB3 controllers and
PCI busses.

There is no default device layout and PCI devices should be added to
the machine using command line options such as :

  -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.0,addr=0x2
  -netdev bridge,id=net0,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=hostnet0

  -device megasas,id=scsi0,bus=pci.0,addr=0x1
  -drive file=$disk,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2

The last patch explores the possibility of user creatable PHB3
devices. Nothing complex but the QEMU powernv machine models a
baremetal system on which it does not make sense to plug such
controllers. I don't think we should allow it.

Git tree available here for testing, based on David's branch:

	https://github.com/legoater/qemu/tree/phb3-3.0

Thanks,

C.


Changes sinces v1 :

 - removed duplication of macros for the register definitions
 - fixed multi chip support
 - introduced a chip class attribute to create all possible PHB3
   devices
 - introduced property handlers to check the validity of the phb index
   and the chip id
 - explored user creatable PHB3 devices

Benjamin Herrenschmidt (1):
  ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge

Cédric Le Goater (1):
  ppc/pnv: make the PHB3 devices user creatable

 default-configs/ppc64-softmmu.mak   |    1 +
 include/hw/pci-host/pnv_phb3.h      |  163 +++++
 include/hw/pci-host/pnv_phb3_regs.h |  467 +++++++++++++
 include/hw/ppc/pnv.h                |   22 +
 include/hw/ppc/pnv_xscom.h          |    9 +
 include/hw/ppc/xics.h               |    1 +
 hw/intc/xics.c                      |    2 +-
 hw/pci-host/pnv_phb3.c              | 1242 +++++++++++++++++++++++++++++++++++
 hw/pci-host/pnv_phb3_msi.c          |  316 +++++++++
 hw/pci-host/pnv_phb3_pbcq.c         |  358 ++++++++++
 hw/ppc/pnv.c                        |   64 +-
 hw/ppc/pnv_xscom.c                  |    6 +-
 hw/pci-host/Makefile.objs           |    1 +
 13 files changed, 2647 insertions(+), 5 deletions(-)
 create mode 100644 include/hw/pci-host/pnv_phb3.h
 create mode 100644 include/hw/pci-host/pnv_phb3_regs.h
 create mode 100644 hw/pci-host/pnv_phb3.c
 create mode 100644 hw/pci-host/pnv_phb3_msi.c
 create mode 100644 hw/pci-host/pnv_phb3_pbcq.c

-- 
2.13.6

^ permalink raw reply	[flat|nested] 29+ messages in thread

* [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-06-28  8:36 [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge Cédric Le Goater
@ 2018-06-28  8:36 ` Cédric Le Goater
  2018-07-09  7:22   ` Cédric Le Goater
                     ` (2 more replies)
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 2/2] ppc/pnv: make the PHB3 devices user creatable Cédric Le Goater
  2018-07-01 18:33 ` [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge no-reply
  2 siblings, 3 replies; 29+ messages in thread
From: Cédric Le Goater @ 2018-06-28  8:36 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin, Cédric Le Goater

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>

This is a model of the PCIe Host Bridge (PHB3) found on a Power8
processor. It includes the PowerBus logic interface (PBCQ), IOMMU
support, PCIe root complex, the XICS MSI and LSI interrupt sources.

The Power8 processor comes in different flavors: Venice, Murano,
Naple, each having a different number of PHBs. All are initialized but
the machine only needs one PCI bus to plug a couple of devices (net,
storage, usbs). The other busses are provided to test complex
configurations. Some platforms, like the Firestone, can also couple
PHBs on the first chip to provide more bandwidth but this is too
specific to model in QEMU.

No default device layout is provided and PCI devices can be added on
any of the available PCI busses (pci.0..2 on a Power8 chip) using
command line options such as :

  -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.0,addr=0x2
  -netdev bridge,id=net0,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=hostnet0

  -device megasas,id=scsi0,bus=pci.0,addr=0x1
  -drive file=$disk,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2

Multi chip is supported, each chip adding its set of PHB3 controllers
and its PCI busses. The model doesn't emulate the EEH error handling
(and may never do).

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
[clg: rewrote the QOM models
      misc fixes
      lots of love and care]
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 default-configs/ppc64-softmmu.mak   |    1 +
 include/hw/pci-host/pnv_phb3.h      |  161 +++++
 include/hw/pci-host/pnv_phb3_regs.h |  467 ++++++++++++++
 include/hw/ppc/pnv.h                |   22 +
 include/hw/ppc/pnv_xscom.h          |    9 +
 include/hw/ppc/xics.h               |    1 +
 hw/intc/xics.c                      |    2 +-
 hw/pci-host/pnv_phb3.c              | 1219 +++++++++++++++++++++++++++++++++++
 hw/pci-host/pnv_phb3_msi.c          |  316 +++++++++
 hw/pci-host/pnv_phb3_pbcq.c         |  347 ++++++++++
 hw/ppc/pnv.c                        |   75 ++-
 hw/ppc/pnv_xscom.c                  |    6 +-
 hw/pci-host/Makefile.objs           |    1 +
 13 files changed, 2622 insertions(+), 5 deletions(-)
 create mode 100644 include/hw/pci-host/pnv_phb3.h
 create mode 100644 include/hw/pci-host/pnv_phb3_regs.h
 create mode 100644 hw/pci-host/pnv_phb3.c
 create mode 100644 hw/pci-host/pnv_phb3_msi.c
 create mode 100644 hw/pci-host/pnv_phb3_pbcq.c

diff --git a/default-configs/ppc64-softmmu.mak b/default-configs/ppc64-softmmu.mak
index b94af6c7c62a..deebba2b044a 100644
--- a/default-configs/ppc64-softmmu.mak
+++ b/default-configs/ppc64-softmmu.mak
@@ -9,6 +9,7 @@ CONFIG_IPMI=y
 CONFIG_IPMI_LOCAL=y
 CONFIG_IPMI_EXTERN=y
 CONFIG_ISA_IPMI_BT=y
+CONFIG_PCIE_PORT=y
 
 # For pSeries
 CONFIG_PSERIES=y
diff --git a/include/hw/pci-host/pnv_phb3.h b/include/hw/pci-host/pnv_phb3.h
new file mode 100644
index 000000000000..459994b751ca
--- /dev/null
+++ b/include/hw/pci-host/pnv_phb3.h
@@ -0,0 +1,161 @@
+/*
+ * QEMU PowerPC PowerNV PHB3 model
+ *
+ * Copyright (c) 2014-2018, IBM Corporation.
+ *
+ * This code is licensed under the GPL version 2 or later. See the
+ * COPYING file in the top-level directory.
+ */
+
+#ifndef PCI_HOST_PNV_PHB3_H
+#define PCI_HOST_PNV_PHB3_H
+
+#include "hw/pci/pci_host.h"
+#include "hw/ppc/xics.h"
+
+typedef struct PnvPHB3 PnvPHB3;
+typedef struct PnvChip PnvChip;
+
+/*
+ * PHB3 XICS Source for MSIs
+ */
+#define TYPE_PHB3_MSI "phb3-msi"
+#define PHB3_MSI(obj) OBJECT_CHECK(Phb3MsiState, (obj), TYPE_PHB3_MSI)
+
+#define PHB3_MAX_MSI     2048
+
+typedef struct Phb3MsiState {
+    ICSState ics;
+
+    PnvPHB3 *phb;
+    uint64_t rba[PHB3_MAX_MSI / 64];
+    uint32_t rba_sum;
+} Phb3MsiState;
+
+void pnv_phb3_msi_update_config(Phb3MsiState *msis, uint32_t base,
+                                uint32_t count);
+void pnv_phb3_msi_send(Phb3MsiState *msis, uint64_t addr, uint16_t data,
+                       int32_t dev_pe);
+void pnv_phb3_msi_ffi(Phb3MsiState *msis, uint64_t val);
+
+
+/* We have one such address space wrapper per possible device
+ * under the PHB since they need to be assigned statically at
+ * qemu device creation time. The relationship to a PE is done
+ * later dynamically. This means we can potentially create a lot
+ * of these guys. Q35 stores them as some kind of radix tree but
+ * we never really need to do fast lookups so instead we simply
+ * keep a QLIST of them for now, we can add the radix if needed
+ * later on.
+ *
+ * We do cache the PE number to speed things up a bit though.
+ */
+typedef struct PnvPhb3DMASpace {
+    PCIBus *bus;
+    uint8_t devfn;
+    int pe_num;         /* Cached PE number */
+#define PHB_INVALID_PE (-1)
+    PnvPHB3 *phb;
+    AddressSpace dma_as;
+    IOMMUMemoryRegion dma_mr;
+    MemoryRegion msi32_mr;
+    MemoryRegion msi64_mr;
+    bool msi32_mapped;
+    bool msi64_mapped;
+    QLIST_ENTRY(PnvPhb3DMASpace) list;
+} PnvPhb3DMASpace;
+
+/*
+ * PHB3 Power Bus Common Queue
+ */
+#define TYPE_PNV_PBCQ "pnv-pbcq"
+#define PNV_PBCQ(obj) OBJECT_CHECK(PnvPBCQState, (obj), TYPE_PNV_PBCQ)
+
+typedef struct PnvPBCQState {
+    DeviceState parent;
+
+    uint32_t nest_xbase;
+    uint32_t spci_xbase;
+    uint32_t pci_xbase;
+#define PBCQ_NEST_REGS_COUNT    0x46
+#define PBCQ_PCI_REGS_COUNT     0x15
+#define PBCQ_SPCI_REGS_COUNT    0x5
+
+    uint64_t nest_regs[PBCQ_NEST_REGS_COUNT];
+    uint64_t spci_regs[PBCQ_SPCI_REGS_COUNT];
+    uint64_t pci_regs[PBCQ_PCI_REGS_COUNT];
+    MemoryRegion mmbar0;
+    MemoryRegion mmbar1;
+    MemoryRegion phbbar;
+    bool mmio0_mapped;
+    bool mmio1_mapped;
+    bool phb_mapped;
+    uint64_t mmio0_base;
+    uint64_t mmio0_size;
+    uint64_t mmio1_base;
+    uint64_t mmio1_size;
+    PnvPHB3 *phb;
+
+    MemoryRegion xscom_nest_regs;
+    MemoryRegion xscom_pci_regs;
+    MemoryRegion xscom_spci_regs;
+} PnvPBCQState;
+
+/*
+ * PHB3 PCI Host Bridge for PowerNV machines (POWER8)
+ */
+#define TYPE_PNV_PHB3 "pnv-phb3"
+#define PNV_PHB3(obj) OBJECT_CHECK(PnvPHB3, (obj), TYPE_PNV_PHB3)
+
+#define PNV_PHB3_NUM_M64      16
+#define PNV_PHB3_NUM_REGS     (0x1000 >> 3)
+#define PNV_PHB3_NUM_LSI      8
+#define PNV_PHB3_NUM_PE       256
+
+#define PCI_MMIO_TOTAL_SIZE   (0x1ull << 60)
+
+struct PnvPHB3 {
+    PCIHostState parent_obj;
+
+    uint32_t chip_id;
+    uint32_t phb_id;
+    char bus_path[8];
+
+    uint64_t regs[PNV_PHB3_NUM_REGS];
+    MemoryRegion mr_regs;
+
+    MemoryRegion mr_m32;
+    MemoryRegion mr_m64[PNV_PHB3_NUM_M64];
+    bool regs_mapped;
+    bool m32_mapped;
+    bool m64_mapped[PNV_PHB3_NUM_M64];
+    MemoryRegion pci_mmio;
+    MemoryRegion pci_io;
+
+    uint64_t ioda_LIST[8];
+    uint64_t ioda_LXIVT[8];
+    uint64_t ioda_TVT[512];
+    uint64_t ioda_M64BT[16];
+    uint64_t ioda_MDT[256];
+    uint64_t ioda_PEEV[4];
+
+    uint32_t total_irq;
+    ICSState lsis;
+    Phb3MsiState msis;
+
+    PnvPBCQState pbcq;
+
+    QLIST_HEAD(, PnvPhb3DMASpace) dma_spaces;
+};
+
+#define TYPE_PNV_PHB3_RC "pnv-phb3-rc"
+
+#define TYPE_PNV_PHB3_ROOT_BUS "pnv-phb3-root-bus"
+
+uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size);
+void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size);
+void pnv_phb3_update_regions(PnvPHB3 *phb);
+void pnv_phb3_remap_irqs(PnvPHB3 *phb);
+
+
+#endif /* PCI_HOST_PNV_PHB3_H */
diff --git a/include/hw/pci-host/pnv_phb3_regs.h b/include/hw/pci-host/pnv_phb3_regs.h
new file mode 100644
index 000000000000..4ddbb5559eb5
--- /dev/null
+++ b/include/hw/pci-host/pnv_phb3_regs.h
@@ -0,0 +1,467 @@
+/* Copyright (c) 2013-2018, IBM Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied.
+ *
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef PCI_HOST_PNV_PHB3_REGS_H
+#define PCI_HOST_PNV_PHB3_REGS_H
+
+/*
+ * PBCQ XSCOM registers
+ */
+
+#define PBCQ_NEST_IRSN_COMPARE  0x1a
+#define PBCQ_NEST_IRSN_COMP           PPC_BITMASK(0, 18)
+#define PBCQ_NEST_IRSN_MASK     0x1b
+#define PBCQ_NEST_LSI_SRC_ID    0x1f
+#define   PBCQ_NEST_LSI_SRC           PPC_BITMASK(0, 7)
+#define PBCQ_NEST_REGS_COUNT    0x46
+#define PBCQ_NEST_MMIO_BAR0     0x40
+#define PBCQ_NEST_MMIO_BAR1     0x41
+#define PBCQ_NEST_PHB_BAR       0x42
+#define PBCQ_NEST_MMIO_MASK0    0x43
+#define PBCQ_NEST_MMIO_MASK1    0x44
+#define PBCQ_NEST_BAR_EN        0x45
+#define   PBCQ_NEST_BAR_EN_MMIO0    PPC_BIT(0)
+#define   PBCQ_NEST_BAR_EN_MMIO1    PPC_BIT(1)
+#define   PBCQ_NEST_BAR_EN_PHB      PPC_BIT(2)
+#define   PBCQ_NEST_BAR_EN_IRSN_RX  PPC_BIT(3)
+#define   PBCQ_NEST_BAR_EN_IRSN_TX  PPC_BIT(4)
+
+#define PBCQ_PCI_REGS_COUNT     0x15
+#define PBCQ_PCI_BAR2           0x0b
+
+#define PBCQ_SPCI_REGS_COUNT    0x5
+#define PBCQ_SPCI_ASB_ADDR      0x0
+#define PBCQ_SPCI_ASB_STATUS    0x1
+#define PBCQ_SPCI_ASB_DATA      0x2
+#define PBCQ_SPCI_AIB_CAPP_EN   0x3
+#define PBCQ_SPCI_CAPP_SEC_TMR  0x4
+
+/*
+ * PHB MMIO registers
+ */
+
+/* PHB Fundamental register set A */
+#define PHB_LSI_SOURCE_ID               0x100
+#define   PHB_LSI_SRC_ID                PPC_BITMASK(5, 12)
+#define PHB_DMA_CHAN_STATUS             0x110
+#define   PHB_DMA_CHAN_ANY_ERR          PPC_BIT(27)
+#define   PHB_DMA_CHAN_ANY_ERR1         PPC_BIT(28)
+#define   PHB_DMA_CHAN_ANY_FREEZE       PPC_BIT(29)
+#define PHB_CPU_LOADSTORE_STATUS        0x120
+#define   PHB_CPU_LS_ANY_ERR            PPC_BIT(27)
+#define   PHB_CPU_LS_ANY_ERR1           PPC_BIT(28)
+#define   PHB_CPU_LS_ANY_FREEZE         PPC_BIT(29)
+#define PHB_DMA_MSI_NODE_ID             0x128
+#define   PHB_DMAMSI_NID_FIXED          PPC_BIT(0)
+#define   PHB_DMAMSI_NID_MASK           PPC_BITMASK(24, 31)
+#define   PHB_DMAMSI_NID_LSH            PPC_BITLSHIFT(31)
+#define PHB_CONFIG_DATA                 0x130
+#define PHB_LOCK0                       0x138
+#define PHB_CONFIG_ADDRESS              0x140
+#define   PHB_CA_ENABLE                 PPC_BIT(0)
+#define   PHB_CA_BUS_MASK               PPC_BITMASK(4, 11)
+#define   PHB_CA_BUS_LSH                PPC_BITLSHIFT(11)
+#define   PHB_CA_DEV_MASK               PPC_BITMASK(12, 16)
+#define   PHB_CA_DEV_LSH                PPC_BITLSHIFT(16)
+#define   PHB_CA_FUNC_MASK              PPC_BITMASK(17, 19)
+#define   PHB_CA_FUNC_LSH               PPC_BITLSHIFT(19)
+#define   PHB_CA_REG_MASK               PPC_BITMASK(20, 31)
+#define   PHB_CA_REG_LSH                PPC_BITLSHIFT(31)
+#define   PHB_CA_PE_MASK                PPC_BITMASK(40, 47)
+#define   PHB_CA_PE_LSH                 PPC_BITLSHIFT(47)
+#define PHB_LOCK1                       0x148
+#define PHB_IVT_BAR                     0x150
+#define   PHB_IVT_BAR_ENABLE            PPC_BIT(0)
+#define   PHB_IVT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 48)
+#define   PHB_IVT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(48)
+#define   PHB_IVT_LENGTH_MASK           PPC_BITMASK(52, 63)
+#define   PHB_IVT_LENGTH_ADDRESS_LSH    PPC_BITLSHIFT(63)
+#define PHB_RBA_BAR                     0x158
+#define   PHB_RBA_BAR_ENABLE            PPC_BIT(0)
+#define   PHB_RBA_BASE_ADDRESS_MASK     PPC_BITMASK(14, 55)
+#define   PHB_RBA_BASE_ADDRESS_LSH      PPC_BITLSHIFT(55)
+#define PHB_PHB3_CONFIG                 0x160
+#define   PHB_PHB3C_64B_TCE_EN          PPC_BIT(2)
+#define   PHB_PHB3C_32BIT_MSI_EN        PPC_BIT(8)
+#define   PHB_PHB3C_64BIT_MSI_EN        PPC_BIT(14)
+#define   PHB_PHB3C_M32_EN              PPC_BIT(16)
+#define PHB_RTT_BAR                     0x168
+#define   PHB_RTT_BAR_ENABLE            PPC_BIT(0)
+#define   PHB_RTT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 46)
+#define   PHB_RTT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(46)
+#define PHB_PELTV_BAR                   0x188
+#define   PHB_PELTV_BAR_ENABLE          PPC_BIT(0)
+#define   PHB_PELTV_BASE_ADDRESS_MASK   PPC_BITMASK(14, 50)
+#define   PHB_PELTV_BASE_ADDRESS_LSH    PPC_BITLSHIFT(50)
+#define PHB_M32_BASE_ADDR               0x190
+#define PHB_M32_BASE_MASK               0x198
+#define PHB_M32_START_ADDR              0x1a0
+#define PHB_PEST_BAR                    0x1a8
+#define   PHB_PEST_BAR_ENABLE           PPC_BIT(0)
+#define   PHB_PEST_BASE_ADDRESS_MASK    PPC_BITMASK(14, 51)
+#define   PHB_PEST_BASE_ADDRESS_LSH     PPC_BITLSHIFT(51)
+#define PHB_M64_UPPER_BITS              0x1f0
+#define PHB_INTREP_TIMER                0x1f8
+#define PHB_DMARD_SYNC                  0x200
+#define PHB_RTC_INVALIDATE              0x208
+#define   PHB_RTC_INVALIDATE_ALL        PPC_BIT(0)
+#define   PHB_RTC_INVALIDATE_RID_MASK   PPC_BITMASK(16, 31)
+#define   PHB_RTC_INVALIDATE_RID_LSH    PPC_BITLSHIFT(31)
+#define PHB_TCE_KILL                    0x210
+#define   PHB_TCE_KILL_ALL              PPC_BIT(0)
+#define PHB_TCE_SPEC_CTL                0x218
+#define PHB_IODA_ADDR                   0x220
+#define   PHB_IODA_AD_AUTOINC           PPC_BIT(0)
+#define   PHB_IODA_AD_TSEL              PPC_BITMASK(11, 15)
+#define   PHB_IODA_AD_TADR              PPC_BITMASK(55, 63)
+#define PHB_IODA_DATA0                  0x228
+#define PHB_FFI_REQUEST                 0x238
+#define   PHB_FFI_LOCK_CLEAR            PPC_BIT(3)
+#define   PHB_FFI_REQUEST_ISN_MASK      PPC_BITMASK(49, 59)
+#define   PHB_FFI_REQUEST_ISN_LSH       PPC_BITLSHIFT(59)
+#define PHB_FFI_LOCK                    0x240
+#define   PHB_FFI_LOCK_STATE            PPC_BIT(0)
+#define PHB_XIVE_UPDATE                 0x248 /* Broken in DD1 */
+#define PHB_PHB3_GEN_CAP                0x250
+#define PHB_PHB3_TCE_CAP                0x258
+#define PHB_PHB3_IRQ_CAP                0x260
+#define PHB_PHB3_EEH_CAP                0x268
+#define PHB_IVC_INVALIDATE              0x2a0
+#define   PHB_IVC_INVALIDATE_ALL        PPC_BIT(0)
+#define   PHB_IVC_INVALIDATE_SID_MASK   PPC_BITMASK(16, 31)
+#define   PHB_IVC_INVALIDATE_SID_LSH    PPC_BITLSHIFT(31)
+#define PHB_IVC_UPDATE                  0x2a8
+#define   PHB_IVC_UPDATE_ENABLE_P       PPC_BIT(0)
+#define   PHB_IVC_UPDATE_ENABLE_Q       PPC_BIT(1)
+#define   PHB_IVC_UPDATE_ENABLE_SERVER  PPC_BIT(2)
+#define   PHB_IVC_UPDATE_ENABLE_PRI     PPC_BIT(3)
+#define   PHB_IVC_UPDATE_ENABLE_GEN     PPC_BIT(4)
+#define   PHB_IVC_UPDATE_ENABLE_CON     PPC_BIT(5)
+#define   PHB_IVC_UPDATE_GEN_MATCH_MASK PPC_BITMASK(6, 7)
+#define   PHB_IVC_UPDATE_GEN_MATCH_LSH  PPC_BITLSHIFT(7)
+#define   PHB_IVC_UPDATE_SERVER_MASK    PPC_BITMASK(8, 23)
+#define   PHB_IVC_UPDATE_SERVER_LSH     PPC_BITLSHIFT(23)
+#define   PHB_IVC_UPDATE_PRI_MASK       PPC_BITMASK(24, 31)
+#define   PHB_IVC_UPDATE_PRI_LSH        PPC_BITLSHIFT(31)
+#define   PHB_IVC_UPDATE_GEN_MASK       PPC_BITMASK(32, 33)
+#define   PHB_IVC_UPDATE_GEN_LSH        PPC_BITLSHIFT(33)
+#define   PHB_IVC_UPDATE_P_MASK         PPC_BITMASK(34, 34)
+#define   PHB_IVC_UPDATE_P_LSH          PPC_BITLSHIFT(34)
+#define   PHB_IVC_UPDATE_Q_MASK         PPC_BITMASK(35, 35)
+#define   PHB_IVC_UPDATE_Q_LSH          PPC_BITLSHIFT(35)
+#define   PHB_IVC_UPDATE_SID_MASK       PPC_BITMASK(48, 63)
+#define   PHB_IVC_UPDATE_SID_LSH        PPC_BITLSHIFT(63)
+#define PHB_PAPR_ERR_INJ_CTL            0x2b0
+#define   PHB_PAPR_ERR_INJ_CTL_INB      PPC_BIT(0)
+#define   PHB_PAPR_ERR_INJ_CTL_OUTB     PPC_BIT(1)
+#define   PHB_PAPR_ERR_INJ_CTL_STICKY   PPC_BIT(2)
+#define   PHB_PAPR_ERR_INJ_CTL_CFG      PPC_BIT(3)
+#define   PHB_PAPR_ERR_INJ_CTL_RD       PPC_BIT(4)
+#define   PHB_PAPR_ERR_INJ_CTL_WR       PPC_BIT(5)
+#define   PHB_PAPR_ERR_INJ_CTL_FREEZE   PPC_BIT(6)
+#define PHB_PAPR_ERR_INJ_ADDR           0x2b8
+#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_MASK       PPC_BITMASK(16, 63)
+#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_LSH        PPC_BITLSHIFT(63)
+#define PHB_PAPR_ERR_INJ_MASK           0x2c0
+#define   PHB_PAPR_ERR_INJ_MASK_CFG_MASK        PPC_BITMASK(4, 11)
+#define   PHB_PAPR_ERR_INJ_MASK_CFG_LSH         PPC_BITLSHIFT(11)
+#define   PHB_PAPR_ERR_INJ_MASK_MMIO_MASK       PPC_BITMASK(16, 63)
+#define   PHB_PAPR_ERR_INJ_MASK_MMIO_LSH        PPC_BITLSHIFT(63)
+#define PHB_ETU_ERR_SUMMARY             0x2c8
+
+/*  UTL registers */
+#define UTL_SYS_BUS_CONTROL             0x400
+#define UTL_STATUS                      0x408
+#define UTL_SYS_BUS_AGENT_STATUS        0x410
+#define UTL_SYS_BUS_AGENT_ERR_SEVERITY  0x418
+#define UTL_SYS_BUS_AGENT_IRQ_EN        0x420
+#define UTL_SYS_BUS_BURST_SZ_CONF       0x440
+#define UTL_REVISION_ID                 0x448
+#define UTL_BCLK_DOMAIN_DBG1            0x460
+#define UTL_BCLK_DOMAIN_DBG2            0x468
+#define UTL_BCLK_DOMAIN_DBG3            0x470
+#define UTL_BCLK_DOMAIN_DBG4            0x478
+#define UTL_BCLK_DOMAIN_DBG5            0x480
+#define UTL_BCLK_DOMAIN_DBG6            0x488
+#define UTL_OUT_POST_HDR_BUF_ALLOC      0x4c0
+#define UTL_OUT_POST_DAT_BUF_ALLOC      0x4d0
+#define UTL_IN_POST_HDR_BUF_ALLOC       0x4e0
+#define UTL_IN_POST_DAT_BUF_ALLOC       0x4f0
+#define UTL_OUT_NP_BUF_ALLOC            0x500
+#define UTL_IN_NP_BUF_ALLOC             0x510
+#define UTL_PCIE_TAGS_ALLOC             0x520
+#define UTL_GBIF_READ_TAGS_ALLOC        0x530
+#define UTL_PCIE_PORT_CONTROL           0x540
+#define UTL_PCIE_PORT_STATUS            0x548
+#define UTL_PCIE_PORT_ERROR_SEV         0x550
+#define UTL_PCIE_PORT_IRQ_EN            0x558
+#define UTL_RC_STATUS                   0x560
+#define UTL_RC_ERR_SEVERITY             0x568
+#define UTL_RC_IRQ_EN                   0x570
+#define UTL_EP_STATUS                   0x578
+#define UTL_EP_ERR_SEVERITY             0x580
+#define UTL_EP_ERR_IRQ_EN               0x588
+#define UTL_PCI_PM_CTRL1                0x590
+#define UTL_PCI_PM_CTRL2                0x598
+#define UTL_GP_CTL1                     0x5a0
+#define UTL_GP_CTL2                     0x5a8
+#define UTL_PCLK_DOMAIN_DBG1            0x5b0
+#define UTL_PCLK_DOMAIN_DBG2            0x5b8
+#define UTL_PCLK_DOMAIN_DBG3            0x5c0
+#define UTL_PCLK_DOMAIN_DBG4            0x5c8
+
+/* PCI-E Stack registers */
+#define PHB_PCIE_SYSTEM_CONFIG          0x600
+#define PHB_PCIE_BUS_NUMBER             0x608
+#define PHB_PCIE_SYSTEM_TEST            0x618
+#define PHB_PCIE_LINK_MANAGEMENT        0x630
+#define   PHB_PCIE_LM_LINK_ACTIVE       PPC_BIT(8)
+#define PHB_PCIE_DLP_TRAIN_CTL          0x640
+#define   PHB_PCIE_DLP_TCTX_DISABLE     PPC_BIT(1)
+#define   PHB_PCIE_DLP_TCRX_DISABLED    PPC_BIT(16)
+#define   PHB_PCIE_DLP_INBAND_PRESENCE  PPC_BIT(19)
+#define   PHB_PCIE_DLP_TC_DL_LINKUP     PPC_BIT(21)
+#define   PHB_PCIE_DLP_TC_DL_PGRESET    PPC_BIT(22)
+#define   PHB_PCIE_DLP_TC_DL_LINKACT    PPC_BIT(23)
+#define PHB_PCIE_SLOP_LOOPBACK_STATUS   0x648
+#define PHB_PCIE_SYS_LINK_INIT          0x668
+#define PHB_PCIE_UTL_CONFIG             0x670
+#define PHB_PCIE_DLP_CONTROL            0x678
+#define PHB_PCIE_UTL_ERRLOG1            0x680
+#define PHB_PCIE_UTL_ERRLOG2            0x688
+#define PHB_PCIE_UTL_ERRLOG3            0x690
+#define PHB_PCIE_UTL_ERRLOG4            0x698
+#define PHB_PCIE_DLP_ERRLOG1            0x6a0
+#define PHB_PCIE_DLP_ERRLOG2            0x6a8
+#define PHB_PCIE_DLP_ERR_STATUS         0x6b0
+#define PHB_PCIE_DLP_ERR_COUNTERS       0x6b8
+#define PHB_PCIE_UTL_ERR_INJECT         0x6c0
+#define PHB_PCIE_TLDLP_ERR_INJECT       0x6c8
+#define PHB_PCIE_LANE_EQ_CNTL0          0x6d0
+#define PHB_PCIE_LANE_EQ_CNTL1          0x6d8
+#define PHB_PCIE_LANE_EQ_CNTL2          0x6e0
+#define PHB_PCIE_LANE_EQ_CNTL3          0x6e8
+#define PHB_PCIE_STRAPPING              0x700
+
+/* Fundamental register set B */
+#define PHB_VERSION                     0x800
+#define PHB_RESET                       0x808
+#define PHB_CONTROL                     0x810
+#define   PHB_CTRL_IVE_128_BYTES        PPC_BIT(24)
+#define PHB_AIB_RX_CRED_INIT_TIMER      0x818
+#define PHB_AIB_RX_CMD_CRED             0x820
+#define PHB_AIB_RX_DATA_CRED            0x828
+#define PHB_AIB_TX_CMD_CRED             0x830
+#define PHB_AIB_TX_DATA_CRED            0x838
+#define PHB_AIB_TX_CHAN_MAPPING         0x840
+#define PHB_AIB_TAG_ENABLE              0x858
+#define PHB_AIB_FENCE_CTRL              0x860
+#define PHB_TCE_TAG_ENABLE              0x868
+#define PHB_TCE_WATERMARK               0x870
+#define PHB_TIMEOUT_CTRL1               0x878
+#define PHB_TIMEOUT_CTRL2               0x880
+#define PHB_QUIESCE_DMA_G               0x888
+#define PHB_AIB_TAG_STATUS              0x900
+#define PHB_TCE_TAG_STATUS              0x908
+
+/* FIR & Error registers */
+#define PHB_LEM_FIR_ACCUM               0xc00
+#define PHB_LEM_FIR_AND_MASK            0xc08
+#define PHB_LEM_FIR_OR_MASK             0xc10
+#define PHB_LEM_ERROR_MASK              0xc18
+#define PHB_LEM_ERROR_AND_MASK          0xc20
+#define PHB_LEM_ERROR_OR_MASK           0xc28
+#define PHB_LEM_ACTION0                 0xc30
+#define PHB_LEM_ACTION1                 0xc38
+#define PHB_LEM_WOF                     0xc40
+#define PHB_ERR_STATUS                  0xc80
+#define PHB_ERR1_STATUS                 0xc88
+#define PHB_ERR_INJECT                  0xc90
+#define PHB_ERR_LEM_ENABLE              0xc98
+#define PHB_ERR_IRQ_ENABLE              0xca0
+#define PHB_ERR_FREEZE_ENABLE           0xca8
+#define PHB_ERR_AIB_FENCE_ENABLE        0xcb0
+#define PHB_ERR_LOG_0                   0xcc0
+#define PHB_ERR_LOG_1                   0xcc8
+#define PHB_ERR_STATUS_MASK             0xcd0
+#define PHB_ERR1_STATUS_MASK            0xcd8
+
+#define PHB_OUT_ERR_STATUS              0xd00
+#define PHB_OUT_ERR1_STATUS             0xd08
+#define PHB_OUT_ERR_INJECT              0xd10
+#define PHB_OUT_ERR_LEM_ENABLE          0xd18
+#define PHB_OUT_ERR_IRQ_ENABLE          0xd20
+#define PHB_OUT_ERR_FREEZE_ENABLE       0xd28
+#define PHB_OUT_ERR_AIB_FENCE_ENABLE    0xd30
+#define PHB_OUT_ERR_LOG_0               0xd40
+#define PHB_OUT_ERR_LOG_1               0xd48
+#define PHB_OUT_ERR_STATUS_MASK         0xd50
+#define PHB_OUT_ERR1_STATUS_MASK        0xd58
+
+#define PHB_INA_ERR_STATUS              0xd80
+#define PHB_INA_ERR1_STATUS             0xd88
+#define PHB_INA_ERR_INJECT              0xd90
+#define PHB_INA_ERR_LEM_ENABLE          0xd98
+#define PHB_INA_ERR_IRQ_ENABLE          0xda0
+#define PHB_INA_ERR_FREEZE_ENABLE       0xda8
+#define PHB_INA_ERR_AIB_FENCE_ENABLE    0xdb0
+#define PHB_INA_ERR_LOG_0               0xdc0
+#define PHB_INA_ERR_LOG_1               0xdc8
+#define PHB_INA_ERR_STATUS_MASK         0xdd0
+#define PHB_INA_ERR1_STATUS_MASK        0xdd8
+
+#define PHB_INB_ERR_STATUS              0xe00
+#define PHB_INB_ERR1_STATUS             0xe08
+#define PHB_INB_ERR_INJECT              0xe10
+#define PHB_INB_ERR_LEM_ENABLE          0xe18
+#define PHB_INB_ERR_IRQ_ENABLE          0xe20
+#define PHB_INB_ERR_FREEZE_ENABLE       0xe28
+#define PHB_INB_ERR_AIB_FENCE_ENABLE    0xe30
+#define PHB_INB_ERR_LOG_0               0xe40
+#define PHB_INB_ERR_LOG_1               0xe48
+#define PHB_INB_ERR_STATUS_MASK         0xe50
+#define PHB_INB_ERR1_STATUS_MASK        0xe58
+
+/* Performance monitor & Debug registers */
+#define PHB_TRACE_CONTROL               0xf80
+#define PHB_PERFMON_CONFIG              0xf88
+#define PHB_PERFMON_CTR0                0xf90
+#define PHB_PERFMON_CTR1                0xf98
+#define PHB_PERFMON_CTR2                0xfa0
+#define PHB_PERFMON_CTR3                0xfa8
+#define PHB_HOTPLUG_OVERRIDE            0xfb0
+#define   PHB_HPOVR_FORCE_RESAMPLE      PPC_BIT(9)
+#define   PHB_HPOVR_PRESENCE_A          PPC_BIT(10)
+#define   PHB_HPOVR_PRESENCE_B          PPC_BIT(11)
+#define   PHB_HPOVR_LINK_ACTIVE         PPC_BIT(12)
+#define   PHB_HPOVR_LINK_BIFURCATED     PPC_BIT(13)
+#define   PHB_HPOVR_LINK_LANE_SWAPPED   PPC_BIT(14)
+
+/*
+ * IODA2 on-chip tables
+ */
+
+#define IODA2_TBL_LIST          1
+#define IODA2_TBL_LXIVT         2
+#define IODA2_TBL_IVC_CAM       3
+#define IODA2_TBL_RBA           4
+#define IODA2_TBL_RCAM          5
+#define IODA2_TBL_MRT           6
+#define IODA2_TBL_PESTA         7
+#define IODA2_TBL_PESTB         8
+#define IODA2_TBL_TVT           9
+#define IODA2_TBL_TCAM          10
+#define IODA2_TBL_TDR           11
+#define IODA2_TBL_M64BT         16
+#define IODA2_TBL_M32DT         17
+#define IODA2_TBL_PEEV          20
+
+/* LXIVT */
+#define IODA2_LXIVT_SERVER              PPC_BITMASK(8, 23)
+#define IODA2_LXIVT_PRIORITY            PPC_BITMASK(24, 31)
+#define IODA2_LXIVT_NODE_ID             PPC_BITMASK(56, 63)
+
+/* IVT */
+#define IODA2_IVT_SERVER                PPC_BITMASK(0, 23)
+#define IODA2_IVT_PRIORITY              PPC_BITMASK(24, 31)
+#define IODA2_IVT_GEN                   PPC_BITMASK(37, 38)
+#define IODA2_IVT_P                     PPC_BITMASK(39, 39)
+#define IODA2_IVT_Q                     PPC_BITMASK(47, 47)
+#define IODA2_IVT_PE                    PPC_BITMASK(48, 63)
+
+/* TVT */
+#define IODA2_TVT_TABLE_ADDR            PPC_BITMASK(0, 47)
+#define IODA2_TVT_NUM_LEVELS            PPC_BITMASK(48, 50)
+#define   IODA2_TVE_1_LEVEL     0
+#define   IODA2_TVE_2_LEVELS    1
+#define   IODA2_TVE_3_LEVELS    2
+#define   IODA2_TVE_4_LEVELS    3
+#define   IODA2_TVE_5_LEVELS    4
+#define IODA2_TVT_TCE_TABLE_SIZE        PPC_BITMASK(51, 55)
+#define IODA2_TVT_IO_PSIZE              PPC_BITMASK(59, 63)
+
+/* PESTA */
+#define IODA2_PESTA_MMIO_FROZEN         PPC_BIT(0)
+
+/* PESTB */
+#define IODA2_PESTB_DMA_STOPPED         PPC_BIT(0)
+
+/* M32DT */
+#define IODA2_M32DT_PE_MASK             PPC_BITMASK(8, 15)
+#define IODA2_M32DT_PE_LSH              PPC_BITLSHIFT(15)
+
+/* M64BT */
+#define IODA2_M64BT_ENABLE              PPC_BIT(0)
+#define IODA2_M64BT_SINGLE_PE           PPC_BIT(1)
+#define IODA2_M64BT_BASE                PPC_BITMASK(2, 31)
+#define IODA2_M64BT_MASK                PPC_BITMASK(34, 63)
+#define IODA2_M64BT_SINGLE_BASE_MASK    PPC_BITMASK(2, 26)
+#define IODA2_M64BT_SINGLE_BASE_LSH     PPC_BITLSHIFT(26)
+#define IODA2_M64BT_PE_HI_MASK          PPC_BITMASK(27, 31)
+#define IODA2_M64BT_PE_HI_LSH           PPC_BITLSHIFT(31)
+#define IODA2_M64BT_SINGLE_MASK_MASK    PPC_BITMASK(34, 58)
+#define IODA2_M64BT_SINGLE_MASK_LSH     PPC_BITLSHIFT(58)
+#define IODA2_M64BT_PE_LOW_MASK         PPC_BITMASK(59, 63)
+#define IODA2_M64BT_PE_LOW_LSH          PPC_BITLSHIFT(63)
+
+/*
+ * IODA2 in-memory tables
+ */
+
+/* PEST
+ *
+ * 2x8 bytes entries, PEST0 and PEST1
+ */
+
+#define IODA2_PEST0_MMIO_CAUSE          PPC_BIT(2)
+#define IODA2_PEST0_CFG_READ            PPC_BIT(3)
+#define IODA2_PEST0_CFG_WRITE           PPC_BIT(4)
+#define IODA2_PEST0_TTYPE_MASK          PPC_BITMASK(5, 7)
+#define IODA2_PEST0_TTYPE_LSH           PPC_BITLSHIFT(7)
+#define   PEST_TTYPE_DMA_WRITE          0
+#define   PEST_TTYPE_MSI                1
+#define   PEST_TTYPE_DMA_READ           2
+#define   PEST_TTYPE_DMA_READ_RESP      3
+#define   PEST_TTYPE_MMIO_LOAD          4
+#define   PEST_TTYPE_MMIO_STORE         5
+#define   PEST_TTYPE_OTHER              7
+#define IODA2_PEST0_CA_RETURN           PPC_BIT(8)
+#define IODA2_PEST0_UTL_RTOS_TIMEOUT    PPC_BIT(8) /* Same bit as CA return */
+#define IODA2_PEST0_UR_RETURN           PPC_BIT(9)
+#define IODA2_PEST0_UTL_NONFATAL        PPC_BIT(10)
+#define IODA2_PEST0_UTL_FATAL           PPC_BIT(11)
+#define IODA2_PEST0_PARITY_UE           PPC_BIT(13)
+#define IODA2_PEST0_UTL_CORRECTABLE     PPC_BIT(14)
+#define IODA2_PEST0_UTL_INTERRUPT       PPC_BIT(15)
+#define IODA2_PEST0_MMIO_XLATE          PPC_BIT(16)
+#define IODA2_PEST0_IODA2_ERROR         PPC_BIT(16) /* Same bit as MMIO xlate */
+#define IODA2_PEST0_TCE_PAGE_FAULT      PPC_BIT(18)
+#define IODA2_PEST0_TCE_ACCESS_FAULT    PPC_BIT(19)
+#define IODA2_PEST0_DMA_RESP_TIMEOUT    PPC_BIT(20)
+#define IODA2_PEST0_AIB_SIZE_INVALID    PPC_BIT(21)
+#define IODA2_PEST0_LEM_BIT_MASK        PPC_BITMASK(26, 31)
+#define IODA2_PEST0_LEM_BIT_LSH         PPC_BITLSHIFT(31)
+#define IODA2_PEST0_RID_MASK            PPC_BITMASK(32, 47)
+#define IODA2_PEST0_RID_LSH             PPC_BITLSHIFT(47)
+#define IODA2_PEST0_MSI_DATA_MASK       PPC_BITMASK(48, 63)
+#define IODA2_PEST0_MSI_DATA_LSH        PPC_BITLSHIFT(63)
+
+#define IODA2_PEST1_FAIL_ADDR_MASK      PPC_BITMASK(3, 63)
+#define IODA2_PEST1_FAIL_ADDR_LSH       PPC_BITLSHIFT(63)
+
+
+#endif /* PCI_HOST_PNV_PHB3_REGS_H */
diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
index 86d5f54e5459..61af65fc0b76 100644
--- a/include/hw/ppc/pnv.h
+++ b/include/hw/ppc/pnv.h
@@ -25,6 +25,7 @@
 #include "hw/ppc/pnv_lpc.h"
 #include "hw/ppc/pnv_psi.h"
 #include "hw/ppc/pnv_occ.h"
+#include "hw/pci-host/pnv_phb3.h"
 
 #define TYPE_PNV_CHIP "pnv-chip"
 #define PNV_CHIP(obj) OBJECT_CHECK(PnvChip, (obj), TYPE_PNV_CHIP)
@@ -72,6 +73,9 @@ typedef struct Pnv8Chip {
     PnvLpcController lpc;
     PnvPsi       psi;
     PnvOCC       occ;
+
+#define PNV8_CHIP_PHB3_MAX 4
+    PnvPHB3      phbs[PNV8_CHIP_PHB3_MAX];
 } Pnv8Chip;
 
 #define TYPE_PNV9_CHIP "pnv9-chip"
@@ -92,6 +96,7 @@ typedef struct PnvChipClass {
     PnvChipType  chip_type;
     uint64_t     chip_cfam_id;
     uint64_t     cores_mask;
+    uint32_t     num_phbs;
 
     hwaddr       xscom_base;
 
@@ -171,6 +176,23 @@ static inline bool pnv_is_power9(PnvMachineState *pnv)
     return pnv_chip_is_power9(pnv->chips[0]);
 }
 
+/*
+ * This is used by devices created on the command line to find a chip
+ * on which to attach to.
+ */
+static inline PnvChip *pnv_get_chip(PnvMachineState *pnv, uint chip_id)
+{
+    int i;
+
+    for (i = 0; i < pnv->num_chips; i++) {
+        PnvChip *chip = pnv->chips[i];
+        if (chip->chip_id == chip_id) {
+            return chip;
+        }
+    }
+    return NULL;
+}
+
 #define PNV_FDT_ADDR          0x01000000
 #define PNV_TIMEBASE_FREQ     512000000ULL
 
diff --git a/include/hw/ppc/pnv_xscom.h b/include/hw/ppc/pnv_xscom.h
index 255b26a5aaf6..e7ebffa5cf8a 100644
--- a/include/hw/ppc/pnv_xscom.h
+++ b/include/hw/ppc/pnv_xscom.h
@@ -73,6 +73,15 @@ typedef struct PnvXScomInterfaceClass {
 #define PNV_XSCOM_OCC_BASE        0x0066000
 #define PNV_XSCOM_OCC_SIZE        0x6000
 
+#define PNV_XSCOM_PBCQ_NEST_BASE  0x2012000
+#define PNV_XSCOM_PBCQ_NEST_SIZE  0x46
+
+#define PNV_XSCOM_PBCQ_PCI_BASE   0x9012000
+#define PNV_XSCOM_PBCQ_PCI_SIZE   0x15
+
+#define PNV_XSCOM_PBCQ_SPCI_BASE  0x9013c00
+#define PNV_XSCOM_PBCQ_SPCI_SIZE  0x5
+
 extern void pnv_xscom_realize(PnvChip *chip, Error **errp);
 extern int pnv_dt_xscom(PnvChip *chip, void *fdt, int offset);
 
diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
index 6ac8a9392da6..966a996c2eac 100644
--- a/include/hw/ppc/xics.h
+++ b/include/hw/ppc/xics.h
@@ -194,6 +194,7 @@ void icp_set_mfrr(ICPState *icp, uint8_t mfrr);
 uint32_t icp_accept(ICPState *ss);
 uint32_t icp_ipoll(ICPState *ss, uint32_t *mfrr);
 void icp_eoi(ICPState *icp, uint32_t xirr);
+void icp_irq(ICSState *ics, int server, int nr, uint8_t priority);
 
 void ics_simple_write_xive(ICSState *ics, int nr, int server,
                            uint8_t priority, uint8_t saved_priority);
diff --git a/hw/intc/xics.c b/hw/intc/xics.c
index b9f1a3c97214..59e2a5217dcc 100644
--- a/hw/intc/xics.c
+++ b/hw/intc/xics.c
@@ -230,7 +230,7 @@ void icp_eoi(ICPState *icp, uint32_t xirr)
     }
 }
 
-static void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
+void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
 {
     ICPState *icp = xics_icp_get(ics->xics, server);
 
diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
new file mode 100644
index 000000000000..e11993e0e1fb
--- /dev/null
+++ b/hw/pci-host/pnv_phb3.c
@@ -0,0 +1,1219 @@
+/*
+ * QEMU PowerPC PowerNV PHB3 model
+ *
+ * Copyright (c) 2014-2018, IBM Corporation.
+ *
+ * This code is licensed under the GPL version 2 or later. See the
+ * COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "qapi/visitor.h"
+#include "qapi/error.h"
+#include "qemu-common.h"
+#include "target/ppc/cpu.h"
+#include "hw/pci-host/pnv_phb3_regs.h"
+#include "hw/pci-host/pnv_phb3.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_bus.h"
+#include "hw/pci/pcie_port.h"
+#include "hw/ppc/pnv.h"
+
+static PCIDevice *pnv_phb3_find_cfg_dev(PnvPHB3 *phb)
+{
+    PCIHostState *pci = PCI_HOST_BRIDGE(phb);
+    uint64_t addr = phb->regs[PHB_CONFIG_ADDRESS >> 3];
+    uint8_t bus, devfn;
+
+    if (!(addr >> 63)) {
+        return NULL;
+    }
+    bus = (addr >> 52) & 0xff;
+    devfn = (addr >> 44) & 0xff;
+
+    return pci_find_device(pci->bus, bus, devfn);
+}
+
+/*
+ * The CONFIG_DATA register expects little endian accesses, but as the
+ * region is big endian, we have to swap the value.
+ */
+static void pnv_phb3_config_write(PnvPHB3 *phb, unsigned off,
+                                  unsigned size, uint64_t val)
+{
+    uint32_t cfg_addr, limit;
+    PCIDevice *pdev;
+
+    pdev = pnv_phb3_find_cfg_dev(phb);
+    if (!pdev) {
+        return;
+    }
+    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xfff;
+    cfg_addr |= off;
+    limit = pci_config_size(pdev);
+    if (limit <= cfg_addr) {
+        /* conventional pci device can be behind pcie-to-pci bridge.
+           256 <= addr < 4K has no effects. */
+        return;
+    }
+    switch (size) {
+    case 1:
+        break;
+    case 2:
+        val = bswap16(val);
+        break;
+    case 4:
+        val = bswap32(val);
+        break;
+    default:
+        g_assert_not_reached();
+    }
+    pci_host_config_write_common(pdev, cfg_addr, limit, val, size);
+}
+
+static uint64_t pnv_phb3_config_read(PnvPHB3 *phb, unsigned off,
+                                     unsigned size)
+{
+    uint32_t cfg_addr, limit;
+    PCIDevice *pdev;
+    uint64_t val;
+
+    pdev = pnv_phb3_find_cfg_dev(phb);
+    if (!pdev) {
+        return ~0ull;
+    }
+    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xffc;
+    cfg_addr |= off;
+    limit = pci_config_size(pdev);
+    if (limit <= cfg_addr) {
+        /* conventional pci device can be behind pcie-to-pci bridge.
+           256 <= addr < 4K has no effects. */
+        return ~0ull;
+    }
+    val = pci_host_config_read_common(pdev, cfg_addr, limit, size);
+    switch (size) {
+    case 1:
+        return val;
+    case 2:
+        return bswap16(val);
+    case 4:
+        return bswap32(val);
+    default:
+        g_assert_not_reached();
+    }
+}
+
+static void pnv_phb3_check_m32(PnvPHB3 *phb)
+{
+    uint64_t base, start, size;
+    MemoryRegion *parent;
+    PnvPBCQState *pbcq = &phb->pbcq;
+
+    if (phb->m32_mapped) {
+        memory_region_del_subregion(phb->mr_m32.container, &phb->mr_m32);
+        phb->m32_mapped = false;
+    }
+
+    /* Disabled ? move on with life ... */
+    if (!(phb->regs[PHB_PHB3_CONFIG >> 3] & PHB_PHB3C_M32_EN)) {
+        return;
+    }
+
+    /* Grab geometry from registers */
+    base = phb->regs[PHB_M32_BASE_ADDR >> 3];
+    start = phb->regs[PHB_M32_START_ADDR >> 3];
+    size = ~(phb->regs[PHB_M32_BASE_MASK >> 3] | 0xfffc000000000000ull) + 1;
+
+    /* Check if it matches an enabled MMIO region in the PBCQ */
+    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
+        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
+        parent = &pbcq->mmbar0;
+        base -= pbcq->mmio0_base;
+    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
+        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
+        parent = &pbcq->mmbar1;
+        base -= pbcq->mmio1_base;
+    } else {
+        return;
+    }
+
+    /* Create alias */
+    memory_region_init_alias(&phb->mr_m32, OBJECT(phb), "phb3-m32",
+                             &phb->pci_mmio, start, size);
+    memory_region_add_subregion(parent, base, &phb->mr_m32);
+    phb->m32_mapped = true;
+}
+
+static void pnv_phb3_check_m64(PnvPHB3 *phb, uint32_t index)
+{
+    uint64_t base, start, size, m64;
+    MemoryRegion *parent;
+    PnvPBCQState *pbcq = &phb->pbcq;
+
+    if (phb->m64_mapped[index]) {
+        /* Should we destroy it in RCU friendly way... ? */
+        memory_region_del_subregion(phb->mr_m64[index].container,
+                                    &phb->mr_m64[index]);
+        phb->m64_mapped[index] = false;
+    }
+
+    /* Get table entry */
+    m64 = phb->ioda_M64BT[index];
+
+    /* Disabled ? move on with life ... */
+    if (!(m64 & IODA2_M64BT_ENABLE)) {
+        return;
+    }
+
+    /* Grab geometry from registers */
+    base = GETFIELD(IODA2_M64BT_BASE, m64) << 20;
+    if (m64 & IODA2_M64BT_SINGLE_PE) {
+        base &= ~0x1ffffffull;
+    }
+    size = GETFIELD(IODA2_M64BT_MASK, m64) << 20;
+    size |= 0xfffc000000000000ull;
+    size = ~size + 1;
+    start = base | (phb->regs[PHB_M64_UPPER_BITS >> 3]);
+
+    /* Check if it matches an enabled MMIO region in the PBCQ */
+    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
+        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
+        parent = &pbcq->mmbar0;
+        base -= pbcq->mmio0_base;
+    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
+        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
+        parent = &pbcq->mmbar1;
+        base -= pbcq->mmio1_base;
+    } else {
+        return;
+    }
+
+    /* Create alias */
+    memory_region_init_alias(&phb->mr_m64[index], OBJECT(phb), "phb3-m64",
+                             &phb->pci_mmio, start, size);
+    memory_region_add_subregion(parent, base, &phb->mr_m64[index]);
+    phb->m64_mapped[index] = true;
+}
+
+static void pnv_phb3_check_all_m64s(PnvPHB3 *phb)
+{
+    uint64_t i;
+
+    for (i = 0; i < PNV_PHB3_NUM_M64; i++) {
+        pnv_phb3_check_m64(phb, i);
+    }
+}
+
+static void pnv_phb3_lxivt_write(PnvPHB3 *phb, unsigned idx, uint64_t val)
+{
+    ICSState *ics = &phb->lsis;
+    uint8_t server, prio;
+
+    phb->ioda_LXIVT[idx] = val & (IODA2_LXIVT_SERVER |
+                                  IODA2_LXIVT_PRIORITY |
+                                  IODA2_LXIVT_NODE_ID);
+    server = GETFIELD(IODA2_LXIVT_SERVER, val);
+    prio = GETFIELD(IODA2_LXIVT_PRIORITY, val);
+
+    /*
+     * The low order 2 bits are the link pointer (Type II interrupts).
+     * Shift back to get a valid IRQ server.
+     */
+    server >>= 2;
+
+    ics_simple_write_xive(ics, idx, server, prio, prio);
+}
+
+static uint64_t *pnv_phb3_ioda_access(PnvPHB3 *phb,
+                                      unsigned *out_table, unsigned *out_idx)
+{
+    uint64_t adreg = phb->regs[PHB_IODA_ADDR >> 3];
+    unsigned int index = GETFIELD(PHB_IODA_AD_TADR, adreg);
+    unsigned int table = GETFIELD(PHB_IODA_AD_TSEL, adreg);
+    unsigned int mask;
+    uint64_t *tptr = NULL;
+
+    switch (table) {
+    case IODA2_TBL_LIST:
+        tptr = phb->ioda_LIST;
+        mask = 7;
+        break;
+    case IODA2_TBL_LXIVT:
+        tptr = phb->ioda_LXIVT;
+        mask = 7;
+        break;
+    case IODA2_TBL_IVC_CAM:
+    case IODA2_TBL_RBA:
+        mask = 31;
+        break;
+    case IODA2_TBL_RCAM:
+        mask = 63;
+        break;
+    case IODA2_TBL_MRT:
+        mask = 7;
+        break;
+    case IODA2_TBL_PESTA:
+    case IODA2_TBL_PESTB:
+        mask = 255;
+        break;
+    case IODA2_TBL_TVT:
+        tptr = phb->ioda_TVT;
+        mask = 511;
+        break;
+    case IODA2_TBL_TCAM:
+    case IODA2_TBL_TDR:
+        mask = 63;
+        break;
+    case IODA2_TBL_M64BT:
+        tptr = phb->ioda_M64BT;
+        mask = 15;
+        break;
+    case IODA2_TBL_M32DT:
+        tptr = phb->ioda_MDT;
+        mask = 255;
+        break;
+    case IODA2_TBL_PEEV:
+        tptr = phb->ioda_PEEV;
+        mask = 3;
+        break;
+    default:
+        return NULL;
+    }
+    index &= mask;
+    if (out_idx) {
+        *out_idx = index;
+    }
+    if (out_table) {
+        *out_table = table;
+    }
+    if (adreg & PHB_IODA_AD_AUTOINC) {
+        index = (index + 1) & mask;
+        adreg = SETFIELD(PHB_IODA_AD_TADR, adreg, index);
+    }
+    if (tptr) {
+        tptr += index;
+    }
+    phb->regs[PHB_IODA_ADDR >> 3] = adreg;
+    return tptr;
+}
+
+static uint64_t pnv_phb3_ioda_read(PnvPHB3 *phb)
+{
+        unsigned table;
+        uint64_t *tptr;
+
+        tptr = pnv_phb3_ioda_access(phb, &table, NULL);
+        if (!tptr) {
+            /* Return 0 on unsupported tables, not ff's */
+            return 0;
+        }
+        return *tptr;
+}
+
+static void pnv_phb3_ioda_write(PnvPHB3 *phb, uint64_t val)
+{
+        unsigned table, idx;
+        uint64_t *tptr;
+
+        tptr = pnv_phb3_ioda_access(phb, &table, &idx);
+        if (!tptr) {
+            return;
+        }
+
+        /* Handle side effects */
+        switch (table) {
+        case IODA2_TBL_LXIVT:
+            pnv_phb3_lxivt_write(phb, idx, val);
+            break;
+        case IODA2_TBL_M64BT:
+            *tptr = val;
+            pnv_phb3_check_m64(phb, idx);
+            break;
+        default:
+            *tptr = val;
+        }
+}
+
+/* This is called whenever the PHB LSI, MSI source ID register or
+ * the PBCQ irq filters are written.
+ */
+void pnv_phb3_remap_irqs(PnvPHB3 *phb)
+{
+    ICSState *ics = &phb->lsis;
+    uint32_t local, global, count, mask, comp;
+    uint64_t baren;
+    PnvPBCQState *pbcq = &phb->pbcq;
+
+    /* First check if we are enabled. Unlike real HW we don't separate TX and RX
+     * so we enable if both are set
+     */
+    baren = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
+    if (!(baren & PBCQ_NEST_BAR_EN_IRSN_RX) ||
+        !(baren & PBCQ_NEST_BAR_EN_IRSN_TX)) {
+        ics->offset = 0;
+        return;
+    }
+
+    /* Grab local LSI source ID */
+    local = GETFIELD(PHB_LSI_SRC_ID, phb->regs[PHB_LSI_SOURCE_ID >> 3]) << 3;
+
+    /* Grab global one and compare */
+    global = GETFIELD(PBCQ_NEST_LSI_SRC,
+                      pbcq->nest_regs[PBCQ_NEST_LSI_SRC_ID]) << 3;
+    if (global != local) {
+        /* This happens during initialization, let's come back when we
+         * are properly configured
+         */
+        ics->offset = 0;
+        return;
+    }
+
+    /* Get the base on the powerbus */
+    comp = GETFIELD(PBCQ_NEST_IRSN_COMP,
+                    pbcq->nest_regs[PBCQ_NEST_IRSN_COMPARE]);
+    mask = GETFIELD(PBCQ_NEST_IRSN_COMP,
+                    pbcq->nest_regs[PBCQ_NEST_IRSN_MASK]);
+    count = ((~mask) + 1) & 0x7ffff;
+    phb->total_irq = count;
+
+    /* Sanity checks */
+    if ((global + PNV_PHB3_NUM_LSI) > count) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "LSIs out of reach: LSI base=%d total irq=%d",
+                      global, count);
+    }
+
+    if (count > 2048) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "More interrupts than supported: %d", count);
+    }
+
+    if ((comp & mask) != comp) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "IRQ compare bits not in mask: comp=0x%x mask=0x%x",
+                      comp, mask);
+        comp &= mask;
+    }
+    /* Setup LSI offset */
+    ics->offset = comp + global;
+
+    /* Setup MSI offset */
+    pnv_phb3_msi_update_config(&phb->msis, comp, count - PNV_PHB3_NUM_LSI);
+}
+
+static void pnv_phb3_lsi_src_id_write(PnvPHB3 *phb, uint64_t val)
+{
+    /* Sanitize content */
+    val &= PHB_LSI_SRC_ID;
+    phb->regs[PHB_LSI_SOURCE_ID >> 3] = val;
+    pnv_phb3_remap_irqs(phb);
+}
+
+static void pnv_phb3_rtc_invalidate(PnvPHB3 *phb, uint64_t val)
+{
+    PnvPhb3DMASpace *ds;
+
+    /* Always invalidate all for now ... */
+    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
+        ds->pe_num = PHB_INVALID_PE;
+    }
+}
+
+
+static void pnv_phb3_update_msi_regions(PnvPhb3DMASpace *ds)
+{
+    uint64_t cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
+
+    if (cfg & PHB_PHB3C_32BIT_MSI_EN) {
+        if (!ds->msi32_mapped) {
+            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
+                                        0xffff0000, &ds->msi32_mr);
+            ds->msi32_mapped = true;
+        }
+    } else {
+        if (ds->msi32_mapped) {
+            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
+                                        &ds->msi32_mr);
+            ds->msi32_mapped = false;
+        }
+    }
+
+    if (cfg & PHB_PHB3C_64BIT_MSI_EN) {
+        if (!ds->msi64_mapped) {
+            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
+                                        (1ull << 60), &ds->msi64_mr);
+            ds->msi64_mapped = true;
+        }
+    } else {
+        if (ds->msi64_mapped) {
+            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
+                                        &ds->msi64_mr);
+            ds->msi64_mapped = false;
+        }
+    }
+}
+
+static void pnv_phb3_update_all_msi_regions(PnvPHB3 *phb)
+{
+    PnvPhb3DMASpace *ds;
+
+    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
+        pnv_phb3_update_msi_regions(ds);
+    }
+}
+
+void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size)
+{
+    PnvPHB3 *phb = opaque;
+    bool changed;
+
+    /* Special case configuration data */
+    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
+        pnv_phb3_config_write(phb, off & 0x3, size, val);
+        return;
+    }
+
+    /* Other registers are 64-bit only */
+    if (size != 8 || off & 0x7) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
+                      off, size);
+        return;
+    }
+
+    /* Handle masking */
+    switch (off) {
+    case PHB_M64_UPPER_BITS:
+        val &= 0xfffc000000000000ull;
+        break;
+    }
+
+    /* Record whether it changed */
+    changed = phb->regs[off >> 3] != val;
+
+    /* Store in register cache first */
+    phb->regs[off >> 3] = val;
+
+    /* Handle side effects */
+    switch (off) {
+    case PHB_PHB3_CONFIG:
+        if (changed) {
+            pnv_phb3_update_all_msi_regions(phb);
+        }
+        /* fall through */
+    case PHB_M32_BASE_ADDR:
+    case PHB_M32_BASE_MASK:
+    case PHB_M32_START_ADDR:
+        if (changed) {
+            pnv_phb3_check_m32(phb);
+        }
+        break;
+    case PHB_M64_UPPER_BITS:
+        if (changed) {
+            pnv_phb3_check_all_m64s(phb);
+        }
+        break;
+    case PHB_LSI_SOURCE_ID:
+        if (changed) {
+            pnv_phb3_lsi_src_id_write(phb, val);
+        }
+        break;
+
+    /* IODA table accesses */
+    case PHB_IODA_DATA0:
+        pnv_phb3_ioda_write(phb, val);
+        break;
+
+    /* RTC invalidation */
+    case PHB_RTC_INVALIDATE:
+        pnv_phb3_rtc_invalidate(phb, val);
+        break;
+
+    /* FFI request */
+    case PHB_FFI_REQUEST:
+        pnv_phb3_msi_ffi(&phb->msis, val);
+        break;
+
+    /* Silent simple writes */
+    case PHB_CONFIG_ADDRESS:
+    case PHB_IODA_ADDR:
+    case PHB_TCE_KILL:
+    case PHB_TCE_SPEC_CTL:
+    case PHB_PEST_BAR:
+    case PHB_PELTV_BAR:
+    case PHB_RTT_BAR:
+    case PHB_RBA_BAR:
+    case PHB_IVT_BAR:
+    case PHB_FFI_LOCK:
+        break;
+
+    /* Noise on anything else */
+    default:
+        qemu_log_mask(LOG_UNIMP, "reg_write 0x%"PRIx64"=%"PRIx64, off, val);
+    }
+}
+
+uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size)
+{
+    PnvPHB3 *phb = opaque;
+    uint64_t val;
+
+    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
+        return pnv_phb3_config_read(phb, off & 0x3, size);
+    }
+
+    /* Other registers are 64-bit only */
+    if (size != 8 || off & 0x7) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
+                      off, size);
+        return ~0ull;
+    }
+
+    /* Default read from cache */
+    val = phb->regs[off >> 3];
+
+    switch (off) {
+    /* Simulate venice DD2.0 */
+    case PHB_VERSION:
+        return 0x000000a300000005ull;
+
+    /* IODA table accesses */
+    case PHB_IODA_DATA0:
+        return pnv_phb3_ioda_read(phb);
+
+    /* Link training always appears trained */
+    case PHB_PCIE_DLP_TRAIN_CTL:
+        return PHB_PCIE_DLP_INBAND_PRESENCE | PHB_PCIE_DLP_TC_DL_LINKACT;
+
+    /* FFI Lock */
+    case PHB_FFI_LOCK:
+        /* Set lock and return previous value */
+        phb->regs[off >> 3] |= PHB_FFI_LOCK_STATE;
+        return val;
+
+    /* Silent simple reads */
+    case PHB_PHB3_CONFIG:
+    case PHB_M32_BASE_ADDR:
+    case PHB_M32_BASE_MASK:
+    case PHB_M32_START_ADDR:
+    case PHB_CONFIG_ADDRESS:
+    case PHB_IODA_ADDR:
+    case PHB_RTC_INVALIDATE:
+    case PHB_TCE_KILL:
+    case PHB_TCE_SPEC_CTL:
+    case PHB_PEST_BAR:
+    case PHB_PELTV_BAR:
+    case PHB_RTT_BAR:
+    case PHB_RBA_BAR:
+    case PHB_IVT_BAR:
+    case PHB_M64_UPPER_BITS:
+        break;
+
+    /* Noise on anything else */
+    default:
+        qemu_log_mask(LOG_UNIMP, "reg_read 0x%"PRIx64"=%"PRIx64, off, val);
+    }
+    return val;
+}
+
+static const MemoryRegionOps pnv_phb3_reg_ops = {
+    .read = pnv_phb3_reg_read,
+    .write = pnv_phb3_reg_write,
+    .valid.min_access_size = 1,
+    .valid.max_access_size = 8,
+    .impl.min_access_size = 1,
+    .impl.max_access_size = 8,
+    .endianness = DEVICE_BIG_ENDIAN,
+};
+
+static int pnv_phb3_map_irq(PCIDevice *pci_dev, int irq_num)
+{
+    /* Check that out properly ... */
+    return irq_num & 3;
+}
+
+static void pnv_phb3_set_irq(void *opaque, int irq_num, int level)
+{
+    PnvPHB3 *phb = opaque;
+
+    /* LSI only ... */
+    if (irq_num > 3) {
+        qemu_log_mask(LOG_GUEST_ERROR, "Unknown IRQ to set %d", irq_num);
+    }
+    qemu_set_irq(phb->lsis.qirqs[irq_num], level);
+}
+
+static bool pnv_phb3_resolve_pe(PnvPhb3DMASpace *ds)
+{
+    uint64_t rtt, addr;
+    uint16_t rte;
+    int bus_num;
+
+    /* Already resolved ? */
+    if (ds->pe_num != PHB_INVALID_PE) {
+        return true;
+    }
+
+    /* We need to lookup the RTT */
+    rtt = ds->phb->regs[PHB_RTT_BAR >> 3];
+    if (!(rtt & PHB_RBA_BAR_ENABLE)) {
+        qemu_log_mask(LOG_GUEST_ERROR, "DMA with RTT BAR disabled !");
+        /* Set error bits ? fence ? ... */
+        return false;
+    }
+
+    /* Read RTE */
+    bus_num = pci_bus_num(ds->bus);
+    addr = rtt & PHB_RTT_BASE_ADDRESS_MASK;
+    addr += 2 * ((bus_num << 8) | ds->devfn);
+    if (dma_memory_read(&address_space_memory, addr, &rte, sizeof(rte))) {
+        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read RTT entry at 0x%"PRIx64,
+                      addr);
+        /* Set error bits ? fence ? ... */
+        return false;
+    }
+    rte = be16_to_cpu(rte);
+
+    /* Fail upon reading of invalid PE# */
+    if (rte >= PNV_PHB3_NUM_PE) {
+        qemu_log_mask(LOG_GUEST_ERROR, "RTE for RID 0x%x invalid (%04x)",
+                      ds->devfn, rte);
+        /* Set error bits ? fence ? ... */
+        return false;
+    }
+    ds->pe_num = rte;
+    return true;
+}
+
+static void pnv_phb3_translate_tve(PnvPhb3DMASpace *ds, hwaddr addr,
+                                   bool is_write, uint64_t tve,
+                                   IOMMUTLBEntry *tlb)
+{
+    uint64_t tta = GETFIELD(IODA2_TVT_TABLE_ADDR, tve);
+    int32_t  lev = GETFIELD(IODA2_TVT_NUM_LEVELS, tve);
+    uint32_t tts = GETFIELD(IODA2_TVT_TCE_TABLE_SIZE, tve);
+    uint32_t tps = GETFIELD(IODA2_TVT_IO_PSIZE, tve);
+
+    /* Invalid levels */
+    if (lev > 4) {
+        qemu_log_mask(LOG_GUEST_ERROR, "Invalid #levels in TVE %d", lev);
+        return;
+    }
+
+    /* IO Page Size of 0 means untranslated, else use TCEs */
+    if (tps == 0) {
+        /* We only support non-translate in top window
+         * XXX FIX THAT, Venice/Murano support it on bottom window
+         * above 4G and Naples suports it on everything
+         */
+        if (!(tve & PPC_BIT(51))) {
+            qemu_log_mask(LOG_GUEST_ERROR,
+                          "xlate for invalid non-translate TVE");
+            return;
+        }
+        /* XXX Handle boundaries */
+
+        /* XXX Use 4k pages like q35 ... for now */
+        tlb->iova = addr & 0xfffffffffffff000ull;
+        tlb->translated_addr = addr & 0x0003fffffffff000ull;
+        tlb->addr_mask = 0xfffull;
+        tlb->perm = IOMMU_RW;
+    } else {
+        uint32_t tce_shift, tbl_shift, sh;
+        uint64_t base, taddr, tce, tce_mask;
+
+        /* TVE disabled ? */
+        if (tts == 0) {
+            qemu_log_mask(LOG_GUEST_ERROR, "xlate for invalid translated TVE");
+            return;
+        }
+
+        /* Address bits per bottom level TCE entry */
+        tce_shift = tps + 11;
+
+        /* Address bits per table level */
+        tbl_shift = tts + 8;
+
+        /* Top level table base address */
+        base = tta << 12;
+
+        /* Total shift to first level */
+        sh = tbl_shift * lev + tce_shift;
+
+        /* XXX Multi-level untested */
+        while ((lev--) >= 0) {
+            /* Grab the TCE address */
+            taddr = base | (((addr >> sh) & ((1ul << tbl_shift) - 1)) << 3);
+            if (dma_memory_read(&address_space_memory, taddr, &tce,
+                                sizeof(tce))) {
+                qemu_log_mask(LOG_GUEST_ERROR,
+                              "Failed to read TCE at 0x%"PRIx64, taddr);
+                return;
+            }
+            tce = be64_to_cpu(tce);
+
+            /* Check permission for indirect TCE */
+            if ((lev >= 0) && !(tce & 3)) {
+                qemu_log_mask(LOG_GUEST_ERROR,
+                              "Invalid indirect TCE at 0x%"PRIx64, taddr);
+                qemu_log_mask(LOG_GUEST_ERROR,
+                              " xlate %"PRIx64":%c TVE=%"PRIx64,
+                              addr, is_write ? 'W' : 'R', tve);
+                qemu_log_mask(LOG_GUEST_ERROR,
+                              " tta=%"PRIx64" lev=%d tts=%d tps=%d",
+                              tta, lev, tts, tps);
+                return;
+            }
+            sh -= tbl_shift;
+            base = tce & ~0xfffull;
+        }
+
+        /* We exit the loop with TCE being the final TCE */
+        tce_mask = ~((1ull << tce_shift) - 1);
+        tlb->iova = addr & tce_mask;
+        tlb->translated_addr = tce & tce_mask;
+        tlb->addr_mask = ~tce_mask;
+        tlb->perm = tce & 3;
+        if ((is_write & !(tce & 2)) || ((!is_write) && !(tce & 1))) {
+            qemu_log_mask(LOG_GUEST_ERROR, "TCE access fault at 0x%"PRIx64,
+                          taddr);
+            qemu_log_mask(LOG_GUEST_ERROR, " xlate %"PRIx64":%c TVE=%"PRIx64,
+                          addr, is_write ? 'W' : 'R', tve);
+            qemu_log_mask(LOG_GUEST_ERROR,
+                          " tta=%"PRIx64" lev=%d tts=%d tps=%d",
+                          tta, lev, tts, tps);
+        }
+    }
+}
+
+static IOMMUTLBEntry pnv_phb3_translate_iommu(IOMMUMemoryRegion *iommu,
+                                              hwaddr addr,
+                                              IOMMUAccessFlags flag,
+                                              int iommu_idx)
+{
+    PnvPhb3DMASpace *ds = container_of(iommu, PnvPhb3DMASpace, dma_mr);
+    int tve_sel;
+    uint64_t tve, cfg;
+    IOMMUTLBEntry ret = {
+        .target_as = &address_space_memory,
+        .iova = addr,
+        .translated_addr = 0,
+        .addr_mask = ~(hwaddr)0,
+        .perm = IOMMU_NONE,
+    };
+
+    /* Resolve PE# */
+    if (!pnv_phb3_resolve_pe(ds)) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
+                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
+        return ret;
+    }
+
+    /* Check top bits */
+    switch (addr >> 60) {
+    case 00:
+        /* DMA or 32-bit MSI ? */
+        cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
+        if ((cfg & PHB_PHB3C_32BIT_MSI_EN) &&
+            ((addr & 0xffffffffffff0000ull) == 0xffff0000ull)) {
+            qemu_log_mask(LOG_GUEST_ERROR, "xlate on 32-bit MSI region");
+            return ret;
+        }
+        /* Choose TVE XXX Use PHB3 Control Register */
+        tve_sel = (addr >> 59) & 1;
+        tve = ds->phb->ioda_TVT[ds->pe_num * 2 + tve_sel];
+        pnv_phb3_translate_tve(ds, addr, flag & IOMMU_WO, tve, &ret);
+        break;
+    case 01:
+        qemu_log_mask(LOG_GUEST_ERROR, "xlate on 64-bit MSI region");
+        break;
+    default:
+        qemu_log_mask(LOG_GUEST_ERROR, "xlate on unsupported address 0x%"PRIx64,
+                      addr);
+    }
+    return ret;
+}
+
+#define TYPE_PNV_PHB3_IOMMU_MEMORY_REGION "pnv-phb3-iommu-memory-region"
+#define PNV_PHB3_IOMMU_MEMORY_REGION(obj) \
+    OBJECT_CHECK(IOMMUMemoryRegion, (obj), TYPE_PNV_PHB3_IOMMU_MEMORY_REGION)
+
+static void pnv_phb3_iommu_memory_region_class_init(ObjectClass *klass,
+                                                    void *data)
+{
+    IOMMUMemoryRegionClass *imrc = IOMMU_MEMORY_REGION_CLASS(klass);
+
+    imrc->translate = pnv_phb3_translate_iommu;
+}
+
+static const TypeInfo pnv_phb3_iommu_memory_region_info = {
+    .parent = TYPE_IOMMU_MEMORY_REGION,
+    .name = TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
+    .class_init = pnv_phb3_iommu_memory_region_class_init,
+};
+
+/*
+ * MSI/MSIX memory region implementation.
+ * The handler handles both MSI and MSIX.
+ */
+static void pnv_phb3_msi_write(void *opaque, hwaddr addr,
+                               uint64_t data, unsigned size)
+{
+    PnvPhb3DMASpace *ds = opaque;
+
+    /* Resolve PE# */
+    if (!pnv_phb3_resolve_pe(ds)) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
+                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
+        return;
+    }
+
+    pnv_phb3_msi_send(&ds->phb->msis, addr, data, ds->pe_num);
+}
+
+static const MemoryRegionOps pnv_phb3_msi_ops = {
+    /* There is no .read as the read result is undefined by PCI spec */
+    .read = NULL,
+    .write = pnv_phb3_msi_write,
+    .endianness = DEVICE_LITTLE_ENDIAN
+};
+
+static AddressSpace *pnv_phb3_dma_iommu(PCIBus *bus, void *opaque, int devfn)
+{
+    PnvPHB3 *phb = opaque;
+    PnvPhb3DMASpace *ds;
+
+    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
+        if (ds->bus == bus && ds->devfn == devfn) {
+            break;
+        }
+    }
+
+    if (ds == NULL) {
+        ds = g_malloc0(sizeof(PnvPhb3DMASpace));
+        ds->bus = bus;
+        ds->devfn = devfn;
+        ds->pe_num = PHB_INVALID_PE;
+        ds->phb = phb;
+        memory_region_init_iommu(&ds->dma_mr, sizeof(ds->dma_mr),
+                                 TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
+                                 OBJECT(phb), "phb3_iommu", UINT64_MAX);
+        address_space_init(&ds->dma_as, MEMORY_REGION(&ds->dma_mr),
+                           "phb3_iommu");
+        memory_region_init_io(&ds->msi32_mr, OBJECT(phb), &pnv_phb3_msi_ops,
+                              ds, "msi32", 0x10000);
+        memory_region_init_io(&ds->msi64_mr, OBJECT(phb), &pnv_phb3_msi_ops,
+                              ds, "msi64", 0x100000);
+        pnv_phb3_update_msi_regions(ds);
+
+        QLIST_INSERT_HEAD(&phb->dma_spaces, ds, list);
+    }
+    return &ds->dma_as;
+}
+
+static void pnv_phb3_instance_init(Object *obj)
+{
+    PnvPHB3 *phb = PNV_PHB3(obj);
+
+    /* Create LSI source */
+    object_initialize(&phb->lsis, sizeof(phb->lsis), TYPE_ICS_SIMPLE);
+    object_property_add_child(obj, "ics-phb-lsi", OBJECT(&phb->lsis), NULL);
+
+    /* Default init ... will be fixed by HW inits */
+    phb->lsis.offset = 0;
+
+    /* Create MSI source */
+    object_initialize(&phb->msis, sizeof(phb->msis), TYPE_PHB3_MSI);
+    object_property_add_const_link(OBJECT(&phb->msis), "phb", obj,
+                                   &error_abort);
+    object_property_add_child(obj, "ics-phb-msi", OBJECT(&phb->msis), NULL);
+
+    /* Create PBCQ */
+    object_initialize(&phb->pbcq, sizeof(phb->pbcq), TYPE_PNV_PBCQ);
+    object_property_add_const_link(OBJECT(&phb->pbcq), "phb", obj,
+                                   &error_abort);
+    object_property_add_child(obj, "pbcq", OBJECT(&phb->pbcq), NULL);
+
+    QLIST_INIT(&phb->dma_spaces);
+}
+
+/*
+ * This could be done under pnv_pbcq_realize
+ */
+static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
+{
+    PCIHostState *pcih = PCI_HOST_BRIDGE(phb);
+    PCIDevice *brdev;
+    PCIDevice *pdev;
+    PCIBus *parent;
+    uint8_t chassis = phb->chip_id * 4 + phb->phb_id;
+    uint8_t chassis_nr = 128;
+
+    /* Add root complex */
+    pdev = pci_create(pcih->bus, 0, TYPE_PNV_PHB3_RC);
+    object_property_add_child(OBJECT(phb), "phb3-rc", OBJECT(pdev), NULL);
+    qdev_prop_set_uint8(DEVICE(pdev), "chassis", chassis);
+    qdev_prop_set_uint16(DEVICE(pdev), "slot", 1);
+    qdev_init_nofail(DEVICE(pdev));
+
+    /* Setup bus for that chip */
+    parent = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
+
+    brdev = pci_create(parent, 0, "pci-bridge");
+    object_property_add_child(OBJECT(parent), "pci-bridge", OBJECT(brdev),
+                              NULL);
+    qdev_prop_set_uint8(DEVICE(brdev), PCI_BRIDGE_DEV_PROP_CHASSIS_NR,
+                        chassis_nr);
+    qdev_init_nofail(DEVICE(brdev));
+}
+
+static void pnv_phb3_realize(DeviceState *dev, Error **errp)
+{
+    PnvPHB3 *phb = PNV_PHB3(dev);
+    PCIHostState *pci = PCI_HOST_BRIDGE(dev);
+    Object *xics = OBJECT(qdev_get_machine());
+    Error *local_err = NULL;
+    int i;
+
+    memory_region_init(&phb->pci_mmio, OBJECT(phb), "pci-mmio",
+                       PCI_MMIO_TOTAL_SIZE);
+
+    /* PHB3 doesn't support IO space. However, qemu gets very upset if
+     * we don't have an IO region to anchor IO BARs onto so we just
+     * initialize one which we never hook up to anything
+     */
+    memory_region_init(&phb->pci_io, OBJECT(phb), "pci-io", 0x10000);
+
+    memory_region_init_io(&phb->mr_regs, OBJECT(phb), &pnv_phb3_reg_ops, phb,
+                          "phb3-regs", 0x1000);
+
+    object_property_set_int(OBJECT(&phb->lsis), PNV_PHB3_NUM_LSI, "nr-irqs",
+                            &error_abort);
+    object_property_add_const_link(OBJECT(&phb->lsis), "xics", xics,
+                                   &error_abort);
+    object_property_set_bool(OBJECT(&phb->lsis), true, "realized", &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    for (i = 0; i < PNV_PHB3_NUM_LSI; i++) {
+        ics_set_irq_type(&phb->lsis, i, true);
+    }
+
+    object_property_add_const_link(OBJECT(&phb->msis), "xics", xics,
+                                   &error_abort);
+    object_property_set_int(OBJECT(&phb->msis), PHB3_MAX_MSI, "nr-irqs",
+                            &error_abort);
+    object_property_set_bool(OBJECT(&phb->msis), true, "realized", &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    object_property_set_bool(OBJECT(&phb->pbcq), true, "realized", &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    pci->bus = pci_register_root_bus(dev, "phb3-root-bus",
+                                pnv_phb3_set_irq, pnv_phb3_map_irq, phb,
+                                &phb->pci_mmio, &phb->pci_io,
+                                0, 4, TYPE_PNV_PHB3_ROOT_BUS);
+    pci_setup_iommu(pci->bus, pnv_phb3_dma_iommu, phb);
+
+    /* Setup the PCI busses */
+    pnv_phb3_pci_create(phb, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+}
+
+void pnv_phb3_update_regions(PnvPHB3 *phb)
+{
+    PnvPBCQState *pbcq = &phb->pbcq;
+
+    /* Unmap first always */
+    if (phb->regs_mapped) {
+        memory_region_del_subregion(&pbcq->phbbar, &phb->mr_regs);
+        phb->regs_mapped = false;
+    }
+
+    /* Map registers if enabled */
+    if (pbcq->phb_mapped) {
+        /* XXX We should use the PHB BAR 2 register but we don't ... */
+        memory_region_add_subregion(&pbcq->phbbar, 0, &phb->mr_regs);
+        phb->regs_mapped = true;
+    }
+
+    /* Check/update m32 */
+    if (phb->m32_mapped) {
+        pnv_phb3_check_m32(phb);
+    }
+}
+
+static const char *pnv_phb3_root_bus_path(PCIHostState *host_bridge,
+                                          PCIBus *rootbus)
+{
+    PnvPHB3 *phb = PNV_PHB3(host_bridge);
+
+    snprintf(phb->bus_path, sizeof(phb->bus_path), "00%02x:%02x",
+             phb->chip_id, phb->phb_id);
+    return phb->bus_path;
+}
+
+static void pnv_phb3_get_phb_id(Object *obj, Visitor *v, const char *name,
+                              void *opaque, Error **errp)
+{
+    Property *prop = opaque;
+    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
+
+    visit_type_uint32(v, name, ptr, errp);
+}
+
+static void pnv_phb3_set_phb_id(Object *obj, Visitor *v, const char *name,
+                              void *opaque, Error **errp)
+{
+    PnvPHB3 *phb = PNV_PHB3(obj);
+    uint32_t phb_id;
+    Error *local_err = NULL;
+
+    visit_type_uint32(v, name, &phb_id, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    /*
+     * Limit to a maximum of 6 PHBs per chip
+     */
+    if (phb_id >= PNV8_CHIP_PHB3_MAX) {
+        error_setg(errp, "invalid PHB index: '%d'", phb_id);
+        return;
+    }
+
+    phb->phb_id = phb_id;
+}
+
+static const PropertyInfo pnv_phb3_phb_id_propinfo = {
+    .name = "irq",
+    .get = pnv_phb3_get_phb_id,
+    .set = pnv_phb3_set_phb_id,
+};
+
+static void pnv_phb3_get_chip_id(Object *obj, Visitor *v, const char *name,
+                              void *opaque, Error **errp)
+{
+    Property *prop = opaque;
+    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
+
+    visit_type_uint32(v, name, ptr, errp);
+}
+
+static void pnv_phb3_set_chip_id(Object *obj, Visitor *v, const char *name,
+                                 void *opaque, Error **errp)
+{
+    PnvPHB3 *phb = PNV_PHB3(obj);
+    uint32_t chip_id;
+    Error *local_err = NULL;
+    PnvChip *chip;
+
+    visit_type_uint32(v, name, &chip_id, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    /*
+     * Check that the chip owning the device is valid.
+     */
+    chip = pnv_get_chip(PNV_MACHINE(qdev_get_machine()), chip_id);
+    if (!chip) {
+        error_setg(errp, "chip id '%d' is invalid", chip_id);
+        return;
+    }
+
+    phb->chip_id = chip_id;
+}
+
+static const PropertyInfo pnv_phb3_chip_id_propinfo = {
+    .name = "irq",
+    .get = pnv_phb3_get_chip_id,
+    .set = pnv_phb3_set_chip_id,
+};
+
+static Property pnv_phb3_properties[] = {
+    DEFINE_PROP("phb-id", PnvPHB3, phb_id, pnv_phb3_phb_id_propinfo,
+                uint32_t),
+    DEFINE_PROP("chip-id", PnvPHB3, chip_id, pnv_phb3_chip_id_propinfo,
+                uint32_t),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pnv_phb3_class_init(ObjectClass *klass, void *data)
+{
+    PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_CLASS(klass);
+    DeviceClass *dc = DEVICE_CLASS(klass);
+
+    hc->root_bus_path = pnv_phb3_root_bus_path;
+    dc->realize = pnv_phb3_realize;
+    dc->props = pnv_phb3_properties;
+    set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+}
+
+static const TypeInfo pnv_phb3_type_info = {
+    .name          = TYPE_PNV_PHB3,
+    .parent        = TYPE_PCI_HOST_BRIDGE,
+    .instance_size = sizeof(PnvPHB3),
+    .class_init    = pnv_phb3_class_init,
+    .instance_init = pnv_phb3_instance_init,
+};
+
+static void pnv_phb3_root_bus_class_init(ObjectClass *klass, void *data)
+{
+    BusClass *k = BUS_CLASS(klass);
+
+    k->max_dev = 1;
+}
+
+static const TypeInfo pnv_phb3_root_bus_info = {
+    .name = TYPE_PNV_PHB3_ROOT_BUS,
+    .parent = TYPE_PCIE_BUS,
+    .class_init = pnv_phb3_root_bus_class_init,
+};
+
+static void pnv_phb3_rc_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+    PCIERootPortClass *rpc = PCIE_ROOT_PORT_CLASS(klass);
+
+    dc->desc     = "IBM PHB3 PCIE Root Port";
+
+    k->vendor_id = PCI_VENDOR_ID_IBM;
+    k->device_id = 0x03dc;
+    k->revision  = 0;
+    rpc->exp_offset = 0x48;
+    rpc->aer_offset = 0x100;
+}
+
+static const TypeInfo pnv_phb3_rc_info = {
+    .name          = TYPE_PNV_PHB3_RC,
+    .parent        = TYPE_PCIE_ROOT_PORT,
+    .class_init    = pnv_phb3_rc_class_init,
+};
+
+static void pnv_phb3_register_types(void)
+{
+    type_register_static(&pnv_phb3_rc_info);
+    type_register_static(&pnv_phb3_type_info);
+    type_register_static(&pnv_phb3_root_bus_info);
+    type_register_static(&pnv_phb3_iommu_memory_region_info);
+}
+
+type_init(pnv_phb3_register_types)
diff --git a/hw/pci-host/pnv_phb3_msi.c b/hw/pci-host/pnv_phb3_msi.c
new file mode 100644
index 000000000000..36927da25b61
--- /dev/null
+++ b/hw/pci-host/pnv_phb3_msi.c
@@ -0,0 +1,316 @@
+/*
+ * QEMU PowerPC PowerNV PHB3 model
+ *
+ * Copyright (c) 2014-2018, IBM Corporation.
+ *
+ * This code is licensed under the GPL version 2 or later. See the
+ * COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "qapi/error.h"
+#include "qemu-common.h"
+#include "target/ppc/cpu.h"
+#include "hw/pci-host/pnv_phb3_regs.h"
+#include "hw/pci-host/pnv_phb3.h"
+#include "hw/ppc/pnv.h"
+#include "hw/pci/msi.h"
+
+static uint64_t phb3_msi_ive_addr(PnvPHB3 *phb, int srcno)
+{
+    uint64_t ivtbar = phb->regs[PHB_IVT_BAR >> 3];
+    uint64_t phbctl = phb->regs[PHB_CONTROL >> 3];
+
+    if (!(ivtbar & PHB_IVT_BAR_ENABLE)) {
+        qemu_log_mask(LOG_GUEST_ERROR, "Failed access to disable IVT BAR !");
+        return 0;
+    }
+
+    if (srcno >= (ivtbar & PHB_IVT_LENGTH_MASK)) {
+        qemu_log_mask(LOG_GUEST_ERROR, "MSI out of bounds (%d vs  0x%"PRIx64")",
+                      srcno, ivtbar & PHB_IVT_LENGTH_MASK);
+        return 0;
+    }
+
+    ivtbar &= PHB_IVT_BASE_ADDRESS_MASK;
+
+    if (phbctl & PHB_CTRL_IVE_128_BYTES) {
+        return ivtbar + 128 * srcno;
+    } else {
+        return ivtbar + 16 * srcno;
+    }
+}
+
+static bool phb3_msi_read_ive(PnvPHB3 *phb, int srcno, uint64_t *out_ive)
+{
+    uint64_t ive_addr, ive;
+
+    ive_addr = phb3_msi_ive_addr(phb, srcno);
+    if (!ive_addr) {
+        return false;
+    }
+
+    if (dma_memory_read(&address_space_memory, ive_addr, &ive, sizeof(ive))) {
+        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read IVE at 0x%" PRIx64,
+                      ive_addr);
+        return false;
+    }
+    *out_ive = be64_to_cpu(ive);
+
+    return true;
+}
+
+static void phb3_msi_set_p(Phb3MsiState *msi, int srcno, uint8_t gen)
+{
+    uint64_t ive_addr;
+    uint8_t p = 0x01 | (gen << 1);
+
+    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
+    if (!ive_addr) {
+        return;
+    }
+
+    if (dma_memory_write(&address_space_memory, ive_addr + 4, &p, 1)) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "Failed to write IVE (set P) at 0x%" PRIx64, ive_addr);
+    }
+}
+
+static void phb3_msi_set_q(Phb3MsiState *msi, int srcno)
+{
+    uint64_t ive_addr;
+    uint8_t q = 0x01;
+
+    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
+    if (!ive_addr) {
+        return;
+    }
+
+    if (dma_memory_write(&address_space_memory, ive_addr + 5, &q, 1)) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "Failed to write IVE (set Q) at 0x%" PRIx64, ive_addr);
+    }
+}
+
+static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
+{
+    ICSState *ics = ICS_BASE(msi);
+    uint64_t ive;
+    uint64_t server, prio, pq, gen;
+
+    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
+        return;
+    }
+
+    server = GETFIELD(IODA2_IVT_SERVER, ive);
+    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
+    pq = GETFIELD(IODA2_IVT_Q, ive);
+    if (!ignore_p) {
+        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
+    }
+    gen = GETFIELD(IODA2_IVT_GEN, ive);
+
+    /*
+     * The low order 2 bits are the link pointer (Type II interrupts).
+     * Shift back to get a valid IRQ server.
+     */
+    server >>= 2;
+
+    switch (pq) {
+    case 0: /* 00 */
+        if (prio == 0xff) {
+            /* Masked, set Q */
+            phb3_msi_set_q(msi, srcno);
+        } else {
+            /* Enabled, set P and send */
+            phb3_msi_set_p(msi, srcno, gen);
+            icp_irq(ics, server, srcno + ics->offset, prio);
+        }
+        break;
+    case 2: /* 10 */
+        /* Already pending, set Q */
+        phb3_msi_set_q(msi, srcno);
+        break;
+    case 1: /* 01 */
+    case 3: /* 11 */
+    default:
+        /* Just drop stuff if Q already set */
+        break;
+    }
+}
+
+static void phb3_msi_set_irq(void *opaque, int srcno, int val)
+{
+    Phb3MsiState *msi = PHB3_MSI(opaque);
+
+    if (val) {
+        phb3_msi_try_send(msi, srcno, false);
+    }
+}
+
+
+void pnv_phb3_msi_send(Phb3MsiState *msi, uint64_t addr, uint16_t data,
+                       int32_t dev_pe)
+{
+    ICSState *ics = ICS_BASE(msi);
+    uint64_t ive;
+    uint16_t pe;
+    uint32_t src = ((addr >> 4) & 0xffff) | (data & 0x1f);
+
+    if (src >= ics->nr_irqs) {
+        qemu_log_mask(LOG_GUEST_ERROR, "MSI %d out of bounds", src);
+        return;
+    }
+    if (dev_pe >= 0) {
+        if (!phb3_msi_read_ive(msi->phb, src, &ive)) {
+            return;
+        }
+        pe = GETFIELD(IODA2_IVT_PE, ive);
+        if (pe != dev_pe) {
+            qemu_log_mask(LOG_GUEST_ERROR,
+                          "MSI %d send by PE#%d but assigned to PE#%d",
+                          src, dev_pe, pe);
+            return;
+        }
+    }
+    qemu_irq_pulse(ics->qirqs[src]);
+}
+
+void pnv_phb3_msi_ffi(Phb3MsiState *msi, uint64_t val)
+{
+    /* Emit interrupt */
+    pnv_phb3_msi_send(msi, val, 0, -1);
+
+    /* Clear FFI lock */
+    msi->phb->regs[PHB_FFI_LOCK >> 3] = 0;
+}
+
+static void phb3_msi_reject(ICSState *ics, uint32_t nr)
+{
+    Phb3MsiState *msi = PHB3_MSI(ics);
+    unsigned int srcno = nr - ics->offset;
+    unsigned int idx = srcno >> 6;
+    unsigned int bit = 1ull << (srcno & 0x3f);
+
+    assert(srcno < PHB3_MAX_MSI);
+
+    msi->rba[idx] |= bit;
+    msi->rba_sum |= (1u << idx);
+}
+
+static void phb3_msi_resend(ICSState *ics)
+{
+    Phb3MsiState *msi = PHB3_MSI(ics);
+    unsigned int i, j;
+
+    if (msi->rba_sum == 0) {
+        return;
+    }
+
+    for (i = 0; i < 32; i++) {
+        if ((msi->rba_sum & (1u << i)) == 0) {
+            continue;
+        }
+        msi->rba_sum &= ~(1u << i);
+        for (j = 0; j < 64; j++) {
+            if ((msi->rba[i] & (1ull << j)) == 0) {
+                continue;
+            }
+            msi->rba[i] &= ~(1u << j);
+            phb3_msi_try_send(msi, i * 64 + j, true);
+        }
+    }
+}
+
+static void phb3_msi_reset(DeviceState *dev)
+{
+    Phb3MsiState *msi = PHB3_MSI(dev);
+    ICSStateClass *icsc = ICS_BASE_GET_CLASS(dev);
+
+    icsc->parent_reset(dev);
+
+    memset(msi->rba, 0, sizeof(msi->rba));
+    msi->rba_sum = 0;
+}
+
+static void phb3_msi_reset_handler(void *dev)
+{
+    phb3_msi_reset(dev);
+}
+
+void pnv_phb3_msi_update_config(Phb3MsiState *msi, uint32_t base,
+                                uint32_t count)
+{
+    ICSState *ics = ICS_BASE(msi);
+
+    if (count > PHB3_MAX_MSI) {
+        count = PHB3_MAX_MSI;
+    }
+    ics->nr_irqs = count;
+    ics->offset = base;
+}
+
+static void phb3_msi_realize(DeviceState *dev, Error **errp)
+{
+    Phb3MsiState *msi = PHB3_MSI(dev);
+    ICSState *ics = ICS_BASE(msi);
+    ICSStateClass *icsc = ICS_BASE_GET_CLASS(ics);
+    Object *obj;
+    Error *local_err = NULL;
+
+    icsc->parent_realize(dev, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
+    if (!obj) {
+        error_propagate(errp, local_err);
+        error_prepend(errp, "required link 'phb' not found: ");
+        return;
+    }
+    msi->phb = PNV_PHB3(obj);
+
+    ics->qirqs = qemu_allocate_irqs(phb3_msi_set_irq, msi, PHB3_MAX_MSI);
+
+    qemu_register_reset(phb3_msi_reset_handler, dev);
+}
+
+static void phb3_msi_instance_init(Object *obj)
+{
+    ICSState *ics = ICS_BASE(obj);
+
+    /* Will be overriden later */
+    ics->offset = 0;
+}
+
+static void phb3_msi_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    ICSStateClass *isc = ICS_BASE_CLASS(klass);
+
+    device_class_set_parent_realize(dc, phb3_msi_realize,
+                                    &isc->parent_realize);
+    device_class_set_parent_reset(dc, phb3_msi_reset,
+                                  &isc->parent_reset);
+
+    isc->reject = phb3_msi_reject;
+    isc->resend = phb3_msi_resend;
+}
+
+static const TypeInfo phb3_msi_info = {
+    .name = TYPE_PHB3_MSI,
+    .parent = TYPE_ICS_BASE,
+    .instance_size = sizeof(Phb3MsiState),
+    .class_init = phb3_msi_class_init,
+    .class_size = sizeof(ICSStateClass),
+    .instance_init = phb3_msi_instance_init,
+};
+
+static void pnv_phb3_msi_register_types(void)
+{
+    type_register_static(&phb3_msi_info);
+}
+
+type_init(pnv_phb3_msi_register_types)
diff --git a/hw/pci-host/pnv_phb3_pbcq.c b/hw/pci-host/pnv_phb3_pbcq.c
new file mode 100644
index 000000000000..34df3ac684e4
--- /dev/null
+++ b/hw/pci-host/pnv_phb3_pbcq.c
@@ -0,0 +1,347 @@
+/*
+ * QEMU PowerPC PowerNV PHB3 model
+ *
+ * Copyright (c) 2014-2018, IBM Corporation.
+ *
+ * This code is licensed under the GPL version 2 or later. See the
+ * COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu-common.h"
+#include "qemu/log.h"
+#include "target/ppc/cpu.h"
+#include "hw/ppc/fdt.h"
+#include "hw/pci-host/pnv_phb3_regs.h"
+#include "hw/pci-host/pnv_phb3.h"
+#include "hw/ppc/pnv_xscom.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_bus.h"
+
+#include <libfdt.h>
+
+static uint64_t pnv_pbcq_nest_xscom_read(void *opaque, hwaddr addr,
+                                         unsigned size)
+{
+    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
+    uint32_t offset = addr >> 3;
+
+    return pbcq->nest_regs[offset];
+}
+
+static uint64_t pnv_pbcq_pci_xscom_read(void *opaque, hwaddr addr,
+                                        unsigned size)
+{
+    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
+    uint32_t offset = addr >> 3;
+
+    return pbcq->pci_regs[offset];
+}
+
+static uint64_t pnv_pbcq_spci_xscom_read(void *opaque, hwaddr addr,
+                                         unsigned size)
+{
+    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
+    uint32_t offset = addr >> 3;
+
+    if (offset == PBCQ_SPCI_ASB_DATA) {
+        return pnv_phb3_reg_read(pbcq->phb,
+                                 pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR], 8);
+    }
+    return pbcq->spci_regs[offset];
+}
+
+static void pnv_pbcq_update_map(PnvPBCQState *pbcq)
+{
+    uint64_t bar_en = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
+    uint64_t bar, mask, size;
+
+    /*
+     * NOTE: This will really not work well if those are remapped
+     * after the PHB has created its sub regions. We could do better
+     * if we had a way to resize regions but we don't really care
+     * that much in practice as the stuff below really only happens
+     * once early during boot
+     */
+
+    /* Handle unmaps */
+    if (pbcq->mmio0_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
+        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar0);
+        pbcq->mmio0_mapped = false;
+    }
+    if (pbcq->mmio1_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
+        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar1);
+        pbcq->mmio1_mapped = false;
+    }
+    if (pbcq->phb_mapped && !(bar_en & PBCQ_NEST_BAR_EN_PHB)) {
+        memory_region_del_subregion(get_system_memory(), &pbcq->phbbar);
+        pbcq->phb_mapped = false;
+    }
+
+    /* Update PHB */
+    pnv_phb3_update_regions(pbcq->phb);
+
+    /* Handle maps */
+    if (!pbcq->mmio0_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
+        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] >> 14;
+        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0];
+        size = ((~mask) >> 14) + 1;
+        memory_region_init(&pbcq->mmbar0, OBJECT(pbcq), "pbcq-mmio0", size);
+        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar0);
+        pbcq->mmio0_mapped = true;
+        pbcq->mmio0_base = bar;
+        pbcq->mmio0_size = size;
+    }
+    if (!pbcq->mmio1_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
+        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] >> 14;
+        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1];
+        size = ((~mask) >> 14) + 1;
+        memory_region_init(&pbcq->mmbar1, OBJECT(pbcq), "pbcq-mmio1", size);
+        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar1);
+        pbcq->mmio1_mapped = true;
+        pbcq->mmio1_base = bar;
+        pbcq->mmio1_size = size;
+    }
+    if (!pbcq->phb_mapped && (bar_en & PBCQ_NEST_BAR_EN_PHB)) {
+        bar = pbcq->nest_regs[PBCQ_NEST_PHB_BAR] >> 14;
+        size = 0x1000;
+        memory_region_init(&pbcq->phbbar, OBJECT(pbcq), "pbcq-phb", size);
+        memory_region_add_subregion(get_system_memory(), bar, &pbcq->phbbar);
+        pbcq->phb_mapped = true;
+    }
+
+    /* Update PHB */
+    pnv_phb3_update_regions(pbcq->phb);
+}
+
+static void pnv_pbcq_nest_xscom_write(void *opaque, hwaddr addr,
+                                uint64_t val, unsigned size)
+{
+    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
+    uint32_t reg = addr >> 3;
+
+    switch (reg) {
+    case PBCQ_NEST_MMIO_BAR0:
+    case PBCQ_NEST_MMIO_BAR1:
+    case PBCQ_NEST_MMIO_MASK0:
+    case PBCQ_NEST_MMIO_MASK1:
+        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] &
+            (PBCQ_NEST_BAR_EN_MMIO0 |
+             PBCQ_NEST_BAR_EN_MMIO1)) {
+            qemu_log_mask(LOG_GUEST_ERROR,
+                           "PHB3: Changing enabled BAR unsupported\n");
+        }
+        pbcq->nest_regs[reg] = val & 0xffffffffc0000000ull;
+        break;
+    case PBCQ_NEST_PHB_BAR:
+        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] & PBCQ_NEST_BAR_EN_PHB) {
+            qemu_log_mask(LOG_GUEST_ERROR,
+                           "PHB3: Changing enabled BAR unsupported\n");
+        }
+        pbcq->nest_regs[reg] = val & 0xfffffffffc000000ull;
+        break;
+    case PBCQ_NEST_BAR_EN:
+        pbcq->nest_regs[reg] = val & 0xf800000000000000ull;
+        pnv_pbcq_update_map(pbcq);
+        pnv_phb3_remap_irqs(pbcq->phb);
+        break;
+    case PBCQ_NEST_IRSN_COMPARE:
+    case PBCQ_NEST_IRSN_MASK:
+        pbcq->nest_regs[reg] = val & PBCQ_NEST_IRSN_COMP;
+        pnv_phb3_remap_irqs(pbcq->phb);
+        break;
+    case PBCQ_NEST_LSI_SRC_ID:
+        pbcq->nest_regs[reg] = val & PBCQ_NEST_LSI_SRC;
+        pnv_phb3_remap_irqs(pbcq->phb);
+        break;
+    }
+
+    /* XXX Don't error out on other regs for now ... */
+}
+
+static void pnv_pbcq_pci_xscom_write(void *opaque, hwaddr addr,
+                                     uint64_t val, unsigned size)
+{
+    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
+    uint32_t reg = addr >> 3;
+
+    switch (reg) {
+    case PBCQ_PCI_BAR2:
+        pbcq->pci_regs[reg] = val & 0xfffffffffc000000ull;
+        pnv_pbcq_update_map(pbcq);
+        break;
+    }
+
+    /* XXX Don't error out on other regs for now ... */
+}
+
+static void pnv_pbcq_spci_xscom_write(void *opaque, hwaddr addr,
+                                uint64_t val, unsigned size)
+{
+    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
+    uint32_t reg = addr >> 3;
+
+    switch (reg) {
+    case PBCQ_SPCI_ASB_ADDR:
+        pbcq->spci_regs[reg] = val & 0xfff;
+        break;
+    case PBCQ_SPCI_ASB_STATUS:
+        pbcq->spci_regs[reg] &= ~val;
+        break;
+    case PBCQ_SPCI_ASB_DATA:
+        pnv_phb3_reg_write(pbcq->phb, pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR],
+                           val, 8);
+        break;
+    case PBCQ_SPCI_AIB_CAPP_EN:
+    case PBCQ_SPCI_CAPP_SEC_TMR:
+        break;
+    }
+
+    /* XXX Don't error out on other regs for now ... */
+}
+
+static const MemoryRegionOps pnv_pbcq_nest_xscom_ops = {
+    .read = pnv_pbcq_nest_xscom_read,
+    .write = pnv_pbcq_nest_xscom_write,
+    .valid.min_access_size = 8,
+    .valid.max_access_size = 8,
+    .impl.min_access_size = 8,
+    .impl.max_access_size = 8,
+    .endianness = DEVICE_BIG_ENDIAN,
+};
+
+static const MemoryRegionOps pnv_pbcq_pci_xscom_ops = {
+    .read = pnv_pbcq_pci_xscom_read,
+    .write = pnv_pbcq_pci_xscom_write,
+    .valid.min_access_size = 8,
+    .valid.max_access_size = 8,
+    .impl.min_access_size = 8,
+    .impl.max_access_size = 8,
+    .endianness = DEVICE_BIG_ENDIAN,
+};
+
+static const MemoryRegionOps pnv_pbcq_spci_xscom_ops = {
+    .read = pnv_pbcq_spci_xscom_read,
+    .write = pnv_pbcq_spci_xscom_write,
+    .valid.min_access_size = 8,
+    .valid.max_access_size = 8,
+    .impl.min_access_size = 8,
+    .impl.max_access_size = 8,
+    .endianness = DEVICE_BIG_ENDIAN,
+};
+
+static void pnv_pbcq_default_bars(PnvPBCQState *pbcq)
+{
+    uint64_t mm0, mm1, reg;
+    PnvPHB3 *phb = pbcq->phb;
+
+    mm0 = 0x3d00000000000ull + 0x4000000000ull * phb->chip_id +
+            0x1000000000ull * phb->phb_id;
+    mm1 = 0x3ff8000000000ull + 0x0200000000ull * phb->chip_id +
+            0x0080000000ull * phb->phb_id;
+    reg = 0x3fffe40000000ull + 0x0000400000ull * phb->chip_id +
+            0x0000100000ull * phb->phb_id;
+
+    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] = mm0 << 14;
+    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] = mm1 << 14;
+    pbcq->nest_regs[PBCQ_NEST_PHB_BAR] = reg << 14;
+    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0] = 0x3fff000000000ull << 14;
+    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1] = 0x3ffff80000000ull << 14;
+    pbcq->pci_regs[PBCQ_PCI_BAR2] = reg << 14;
+}
+
+static void pnv_pbcq_realize(DeviceState *dev, Error **errp)
+{
+    PnvPBCQState *pbcq = PNV_PBCQ(dev);
+    PnvPHB3 *phb;
+    Object *obj;
+    Error *local_err = NULL;
+    char name[32];
+
+    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
+    if (!obj) {
+        error_propagate(errp, local_err);
+        error_prepend(errp, "required link 'phb' not found: ");
+        return;
+    }
+    phb = pbcq->phb = PNV_PHB3(obj);
+
+    /* XXX Fix OPAL to do that: establish default BAR values */
+    pnv_pbcq_default_bars(pbcq);
+
+    /* Initialize the XSCOM region for the PBCQ registers */
+    snprintf(name, sizeof(name), "xscom-pbcq-nest-%d.%d",
+             phb->chip_id, phb->phb_id);
+    pnv_xscom_region_init(&pbcq->xscom_nest_regs, OBJECT(dev),
+                          &pnv_pbcq_nest_xscom_ops, pbcq, name,
+                          PNV_XSCOM_PBCQ_NEST_SIZE);
+    snprintf(name, sizeof(name), "xscom-pbcq-pci-%d.%d",
+             phb->chip_id, phb->phb_id);
+    pnv_xscom_region_init(&pbcq->xscom_pci_regs, OBJECT(dev),
+                          &pnv_pbcq_pci_xscom_ops, pbcq, name,
+                          PNV_XSCOM_PBCQ_PCI_SIZE);
+    snprintf(name, sizeof(name), "xscom-pbcq-spci-%d.%d",
+             phb->chip_id, phb->phb_id);
+    pnv_xscom_region_init(&pbcq->xscom_spci_regs, OBJECT(dev),
+                          &pnv_pbcq_spci_xscom_ops, pbcq, name,
+                          PNV_XSCOM_PBCQ_SPCI_SIZE);
+}
+
+static int pnv_pbcq_dt_xscom(PnvXScomInterface *dev, void *fdt,
+                             int xscom_offset)
+{
+    const char compat[] = "ibm,power8-pbcq";
+    PnvPHB3 *phb = PNV_PBCQ(dev)->phb;
+    char *name;
+    int offset;
+    uint32_t lpc_pcba = PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id;
+    uint32_t reg[] = {
+        cpu_to_be32(lpc_pcba),
+        cpu_to_be32(PNV_XSCOM_PBCQ_NEST_SIZE),
+        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id),
+        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_SIZE),
+        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id),
+        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_SIZE)
+    };
+
+    name = g_strdup_printf("pbcq@%x", lpc_pcba);
+    offset = fdt_add_subnode(fdt, xscom_offset, name);
+    _FDT(offset);
+    g_free(name);
+
+    _FDT((fdt_setprop(fdt, offset, "reg", reg, sizeof(reg))));
+
+    _FDT((fdt_setprop_cell(fdt, offset, "ibm,phb-index", phb->phb_id)));
+    _FDT((fdt_setprop_cell(fdt, offset, "ibm,chip-id", phb->chip_id)));
+    _FDT((fdt_setprop(fdt, offset, "compatible", compat,
+                      sizeof(compat))));
+    return 0;
+}
+
+static void pnv_pbcq_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PnvXScomInterfaceClass *xdc = PNV_XSCOM_INTERFACE_CLASS(klass);
+
+    xdc->dt_xscom = pnv_pbcq_dt_xscom;
+
+    dc->realize = pnv_pbcq_realize;
+}
+
+static const TypeInfo pnv_pbcq_type_info = {
+    .name          = TYPE_PNV_PBCQ,
+    .parent        = TYPE_DEVICE,
+    .instance_size = sizeof(PnvPBCQState),
+    .class_init    = pnv_pbcq_class_init,
+    .interfaces    = (InterfaceInfo[]) {
+        { TYPE_PNV_XSCOM_INTERFACE },
+        { }
+    }
+};
+
+static void pnv_pbcq_register_types(void)
+{
+    type_register_static(&pnv_pbcq_type_info);
+}
+
+type_init(pnv_pbcq_register_types)
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 7401ffe5b01c..77e92d5e2e66 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -44,6 +44,7 @@
 #include "hw/isa/isa.h"
 #include "hw/char/serial.h"
 #include "hw/timer/mc146818rtc.h"
+#include "hw/pci/msi.h"
 
 #include <libfdt.h>
 
@@ -555,6 +556,9 @@ static void pnv_init(MachineState *machine)
     int i;
     char *chip_typename;
 
+    /* MSIs are supported on this platform */
+    msi_nonbroken = true;
+
     /* allocate RAM */
     if (machine->ram_size < (1 * G_BYTE)) {
         warn_report("skiboot may not work with < 1GB of RAM");
@@ -725,6 +729,8 @@ static Object *pnv_chip_power9_intc_create(PnvChip *chip, Object *child,
 static void pnv_chip_power8_instance_init(Object *obj)
 {
     Pnv8Chip *chip8 = PNV8_CHIP(obj);
+    PnvChipClass *pcc = PNV_CHIP_GET_CLASS(obj);
+    int i;
 
     object_initialize(&chip8->psi, sizeof(chip8->psi), TYPE_PNV_PSI);
     object_property_add_child(obj, "psi", OBJECT(&chip8->psi), NULL);
@@ -740,6 +746,12 @@ static void pnv_chip_power8_instance_init(Object *obj)
     object_property_add_child(obj, "occ", OBJECT(&chip8->occ), NULL);
     object_property_add_const_link(OBJECT(&chip8->occ), "psi",
                                    OBJECT(&chip8->psi), &error_abort);
+
+    for (i = 0; i < pcc->num_phbs; i++) {
+        object_initialize(&chip8->phbs[i], sizeof(chip8->phbs[i]),
+                          TYPE_PNV_PHB3);
+        object_property_add_child(obj, "phb[*]", OBJECT(&chip8->phbs[i]), NULL);
+    }
 }
 
 static void pnv_chip_icp_realize(Pnv8Chip *chip8, Error **errp)
@@ -780,6 +792,7 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
     PnvChip *chip = PNV_CHIP(dev);
     Pnv8Chip *chip8 = PNV8_CHIP(dev);
     Error *local_err = NULL;
+    int i;
 
     pcc->parent_realize(dev, &local_err);
     if (local_err) {
@@ -817,6 +830,33 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
         return;
     }
     pnv_xscom_add_subregion(chip, PNV_XSCOM_OCC_BASE, &chip8->occ.xscom_regs);
+
+    /* Create the PHB3 controllers */
+    for (i = 0; i < pcc->num_phbs; i++) {
+        PnvPHB3 *phb = &chip8->phbs[i];
+        PnvPBCQState *pbcq = &phb->pbcq;
+
+        object_property_set_int(OBJECT(phb), i, "phb-id", &error_fatal);
+        object_property_set_int(OBJECT(phb), chip->chip_id, "chip-id",
+                                &error_fatal);
+        object_property_set_bool(OBJECT(phb), true, "realized", &local_err);
+        if (local_err) {
+            error_propagate(errp, local_err);
+            return;
+        }
+        qdev_set_parent_bus(DEVICE(phb), sysbus_get_default());
+
+        /* Populate the XSCOM address space. */
+        pnv_xscom_add_subregion(chip,
+                                PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
+                                &pbcq->xscom_nest_regs);
+        pnv_xscom_add_subregion(chip,
+                                PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
+                                &pbcq->xscom_pci_regs);
+        pnv_xscom_add_subregion(chip,
+                                PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
+                                &pbcq->xscom_spci_regs);
+    }
 }
 
 static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
@@ -827,6 +867,7 @@ static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
     k->chip_type = PNV_CHIP_POWER8E;
     k->chip_cfam_id = 0x221ef04980000000ull;  /* P8 Murano DD2.1 */
     k->cores_mask = POWER8E_CORE_MASK;
+    k->num_phbs = 2;
     k->core_pir = pnv_chip_core_pir_p8;
     k->intc_create = pnv_chip_power8_intc_create;
     k->isa_create = pnv_chip_power8_isa_create;
@@ -845,6 +886,7 @@ static void pnv_chip_power8_class_init(ObjectClass *klass, void *data)
     k->chip_type = PNV_CHIP_POWER8;
     k->chip_cfam_id = 0x220ea04980000000ull; /* P8 Venice DD2.0 */
     k->cores_mask = POWER8_CORE_MASK;
+    k->num_phbs = 3;
     k->core_pir = pnv_chip_core_pir_p8;
     k->intc_create = pnv_chip_power8_intc_create;
     k->isa_create = pnv_chip_power8_isa_create;
@@ -863,6 +905,7 @@ static void pnv_chip_power8nvl_class_init(ObjectClass *klass, void *data)
     k->chip_type = PNV_CHIP_POWER8NVL;
     k->chip_cfam_id = 0x120d304980000000ull;  /* P8 Naples DD1.0 */
     k->cores_mask = POWER8_CORE_MASK;
+    k->num_phbs = 4;
     k->core_pir = pnv_chip_core_pir_p8;
     k->intc_create = pnv_chip_power8_intc_create;
     k->isa_create = pnv_chip_power8nvl_isa_create;
@@ -897,6 +940,7 @@ static void pnv_chip_power9_class_init(ObjectClass *klass, void *data)
     k->chip_type = PNV_CHIP_POWER9;
     k->chip_cfam_id = 0x220d104900008000ull; /* P9 Nimbus DD2.0 */
     k->cores_mask = POWER9_CORE_MASK;
+    k->num_phbs = 0;
     k->core_pir = pnv_chip_core_pir_p9;
     k->intc_create = pnv_chip_power9_intc_create;
     k->isa_create = pnv_chip_power9_isa_create;
@@ -1047,14 +1091,24 @@ static void pnv_chip_class_init(ObjectClass *klass, void *data)
 static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
 {
     PnvMachineState *pnv = PNV_MACHINE(xi);
-    int i;
+    int i, j;
 
     for (i = 0; i < pnv->num_chips; i++) {
+        PnvChip *chip = pnv->chips[i];
         Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
+        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
 
         if (ics_valid_irq(&chip8->psi.ics, irq)) {
             return &chip8->psi.ics;
         }
+        for (j = 0; j < pcc->num_phbs; j++) {
+            if (ics_valid_irq(&chip8->phbs[j].lsis, irq)) {
+                return &chip8->phbs[j].lsis;
+            }
+            if (ics_valid_irq(ICS_BASE(&chip8->phbs[j].msis), irq)) {
+                return ICS_BASE(&chip8->phbs[j].msis);
+            }
+        }
     }
     return NULL;
 }
@@ -1062,11 +1116,18 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
 static void pnv_ics_resend(XICSFabric *xi)
 {
     PnvMachineState *pnv = PNV_MACHINE(xi);
-    int i;
+    int i, j;
 
     for (i = 0; i < pnv->num_chips; i++) {
+        PnvChip *chip = pnv->chips[i];
         Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
+        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
+
         ics_resend(&chip8->psi.ics);
+        for (j = 0; j < pcc->num_phbs; j++) {
+            ics_resend(&chip8->phbs[j].lsis);
+            ics_resend(ICS_BASE(&chip8->phbs[j].msis));
+        }
     }
 }
 
@@ -1097,7 +1158,7 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
                                Monitor *mon)
 {
     PnvMachineState *pnv = PNV_MACHINE(obj);
-    int i;
+    int i, j;
     CPUState *cs;
 
     CPU_FOREACH(cs) {
@@ -1107,8 +1168,16 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
     }
 
     for (i = 0; i < pnv->num_chips; i++) {
+        PnvChip *chip = pnv->chips[i];
         Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
+        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
+
         ics_pic_print_info(&chip8->psi.ics, mon);
+
+        for (j = 0; j < pcc->num_phbs; j++) {
+            ics_pic_print_info(&chip8->phbs[j].lsis, mon);
+            ics_pic_print_info(ICS_BASE(&chip8->phbs[j].msis), mon);
+        }
     }
 }
 
diff --git a/hw/ppc/pnv_xscom.c b/hw/ppc/pnv_xscom.c
index 46fae41f32b0..83f05d06053f 100644
--- a/hw/ppc/pnv_xscom.c
+++ b/hw/ppc/pnv_xscom.c
@@ -252,7 +252,11 @@ int pnv_dt_xscom(PnvChip *chip, void *fdt, int root_offset)
     args.fdt = fdt;
     args.xscom_offset = xscom_offset;
 
-    object_child_foreach(OBJECT(chip), xscom_dt_child, &args);
+    /* Some PnvXScomInterface objects lie a bit deeper (PnvPBCQState)
+     * than the first layer, so we need to loop on the whole object
+     * hierarchy to catch them
+     */
+    object_child_foreach_recursive(OBJECT(chip), xscom_dt_child, &args);
     return 0;
 }
 
diff --git a/hw/pci-host/Makefile.objs b/hw/pci-host/Makefile.objs
index 6d6597c06563..1b5b782bd986 100644
--- a/hw/pci-host/Makefile.objs
+++ b/hw/pci-host/Makefile.objs
@@ -19,3 +19,4 @@ common-obj-$(CONFIG_PCI_GENERIC) += gpex.o
 common-obj-$(CONFIG_PCI_XILINX) += xilinx-pcie.o
 
 common-obj-$(CONFIG_PCI_DESIGNWARE) += designware.o
+obj-$(CONFIG_POWERNV) += pnv_phb3.o pnv_phb3_pbcq.o pnv_phb3_msi.o
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 29+ messages in thread

* [Qemu-devel] [PATCH v2 2/2] ppc/pnv: make the PHB3 devices user creatable
  2018-06-28  8:36 [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge Cédric Le Goater
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 1/2] " Cédric Le Goater
@ 2018-06-28  8:36 ` Cédric Le Goater
  2018-07-01 18:33 ` [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge no-reply
  2 siblings, 0 replies; 29+ messages in thread
From: Cédric Le Goater @ 2018-06-28  8:36 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin, Cédric Le Goater

Extra PHB3 devices and PCI devices can now be added to the powernv
machine using such a command:

  -device pnv-phb3,chip-id=0,index=1,id=phb-0.1 \
  -device nec-usb-xhci,bus=pci.1,addr=0x7

The 'index' property identifies the PHB3 in the chip and 'chip-id' the
chip it belongs to. The PHB3 identifier property name is changed to
'index' to match the attribute name of the pci-root device used by
libvirt for sPAPR machines.

One problem we have to solve when devices are created on the command
line is the object parenthood. The powernv machine uses the object
hierarchy to populate the device tree and each device should be
parented to the chip it belongs to. This is not the case for user
created device which are parented to the container "/unattached". The
changes below make sure PHB3 device is parented to its chip by
reparenting the object if necessary.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/pci-host/pnv_phb3.h |  2 ++
 hw/pci-host/pnv_phb3.c         | 25 ++++++++++++++++++++++++-
 hw/pci-host/pnv_phb3_pbcq.c    | 11 +++++++++++
 hw/ppc/pnv.c                   | 17 +++--------------
 4 files changed, 40 insertions(+), 15 deletions(-)

diff --git a/include/hw/pci-host/pnv_phb3.h b/include/hw/pci-host/pnv_phb3.h
index 459994b751ca..6e328183f9e0 100644
--- a/include/hw/pci-host/pnv_phb3.h
+++ b/include/hw/pci-host/pnv_phb3.h
@@ -146,6 +146,8 @@ struct PnvPHB3 {
     PnvPBCQState pbcq;
 
     QLIST_HEAD(, PnvPhb3DMASpace) dma_spaces;
+
+    PnvChip *chip;
 };
 
 #define TYPE_PNV_PHB3_RC "pnv-phb3-rc"
diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
index e11993e0e1fb..c5e9bd8a23af 100644
--- a/hw/pci-host/pnv_phb3.c
+++ b/hw/pci-host/pnv_phb3.c
@@ -970,6 +970,16 @@ static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
     qdev_init_nofail(DEVICE(brdev));
 }
 
+static void pnv_phb3_parent_fixup(Object *obj, Object *parent, Error **errp)
+{
+    if (obj->parent != parent) {
+        object_ref(obj);
+        object_unparent(obj);
+        object_property_add_child(parent, DEVICE(obj)->id, obj, errp);
+        object_unref(obj);
+    }
+}
+
 static void pnv_phb3_realize(DeviceState *dev, Error **errp)
 {
     PnvPHB3 *phb = PNV_PHB3(dev);
@@ -978,6 +988,17 @@ static void pnv_phb3_realize(DeviceState *dev, Error **errp)
     Error *local_err = NULL;
     int i;
 
+    /*
+     * PHB3 devices created on the command line are not parented to
+     * the chip. Make sure they are because this is necessary to build
+     * correctly the device tree.
+     */
+    pnv_phb3_parent_fixup(OBJECT(phb), OBJECT(phb->chip), &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
     memory_region_init(&phb->pci_mmio, OBJECT(phb), "pci-mmio",
                        PCI_MMIO_TOTAL_SIZE);
 
@@ -1138,6 +1159,7 @@ static void pnv_phb3_set_chip_id(Object *obj, Visitor *v, const char *name,
         return;
     }
 
+    phb->chip = chip;
     phb->chip_id = chip_id;
 }
 
@@ -1148,7 +1170,7 @@ static const PropertyInfo pnv_phb3_chip_id_propinfo = {
 };
 
 static Property pnv_phb3_properties[] = {
-    DEFINE_PROP("phb-id", PnvPHB3, phb_id, pnv_phb3_phb_id_propinfo,
+    DEFINE_PROP("index", PnvPHB3, phb_id, pnv_phb3_phb_id_propinfo,
                 uint32_t),
     DEFINE_PROP("chip-id", PnvPHB3, chip_id, pnv_phb3_chip_id_propinfo,
                 uint32_t),
@@ -1164,6 +1186,7 @@ static void pnv_phb3_class_init(ObjectClass *klass, void *data)
     dc->realize = pnv_phb3_realize;
     dc->props = pnv_phb3_properties;
     set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+    dc->user_creatable = true;
 }
 
 static const TypeInfo pnv_phb3_type_info = {
diff --git a/hw/pci-host/pnv_phb3_pbcq.c b/hw/pci-host/pnv_phb3_pbcq.c
index 34df3ac684e4..403c228a2380 100644
--- a/hw/pci-host/pnv_phb3_pbcq.c
+++ b/hw/pci-host/pnv_phb3_pbcq.c
@@ -285,6 +285,17 @@ static void pnv_pbcq_realize(DeviceState *dev, Error **errp)
     pnv_xscom_region_init(&pbcq->xscom_spci_regs, OBJECT(dev),
                           &pnv_pbcq_spci_xscom_ops, pbcq, name,
                           PNV_XSCOM_PBCQ_SPCI_SIZE);
+
+    /* Populate the XSCOM address space. */
+    pnv_xscom_add_subregion(phb->chip,
+                            PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
+                            &pbcq->xscom_nest_regs);
+    pnv_xscom_add_subregion(phb->chip,
+                            PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
+                            &pbcq->xscom_pci_regs);
+    pnv_xscom_add_subregion(phb->chip,
+                            PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
+                            &pbcq->xscom_spci_regs);
 }
 
 static int pnv_pbcq_dt_xscom(PnvXScomInterface *dev, void *fdt,
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 77e92d5e2e66..83e81b9ae470 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -834,9 +834,8 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
     /* Create the PHB3 controllers */
     for (i = 0; i < pcc->num_phbs; i++) {
         PnvPHB3 *phb = &chip8->phbs[i];
-        PnvPBCQState *pbcq = &phb->pbcq;
 
-        object_property_set_int(OBJECT(phb), i, "phb-id", &error_fatal);
+        object_property_set_int(OBJECT(phb), i, "index", &error_fatal);
         object_property_set_int(OBJECT(phb), chip->chip_id, "chip-id",
                                 &error_fatal);
         object_property_set_bool(OBJECT(phb), true, "realized", &local_err);
@@ -845,17 +844,6 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
             return;
         }
         qdev_set_parent_bus(DEVICE(phb), sysbus_get_default());
-
-        /* Populate the XSCOM address space. */
-        pnv_xscom_add_subregion(chip,
-                                PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
-                                &pbcq->xscom_nest_regs);
-        pnv_xscom_add_subregion(chip,
-                                PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
-                                &pbcq->xscom_pci_regs);
-        pnv_xscom_add_subregion(chip,
-                                PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
-                                &pbcq->xscom_spci_regs);
     }
 }
 
@@ -886,7 +874,7 @@ static void pnv_chip_power8_class_init(ObjectClass *klass, void *data)
     k->chip_type = PNV_CHIP_POWER8;
     k->chip_cfam_id = 0x220ea04980000000ull; /* P8 Venice DD2.0 */
     k->cores_mask = POWER8_CORE_MASK;
-    k->num_phbs = 3;
+    k->num_phbs = 1;
     k->core_pir = pnv_chip_core_pir_p8;
     k->intc_create = pnv_chip_power8_intc_create;
     k->isa_create = pnv_chip_power8_isa_create;
@@ -1248,6 +1236,7 @@ static void pnv_machine_class_init(ObjectClass *oc, void *data)
     xic->ics_get = pnv_ics_get;
     xic->ics_resend = pnv_ics_resend;
     ispc->print_info = pnv_pic_print_info;
+    machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB3);
 
     pnv_machine_class_props_init(oc);
 }
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-06-28  8:36 [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge Cédric Le Goater
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 1/2] " Cédric Le Goater
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 2/2] ppc/pnv: make the PHB3 devices user creatable Cédric Le Goater
@ 2018-07-01 18:33 ` no-reply
  2 siblings, 0 replies; 29+ messages in thread
From: no-reply @ 2018-07-01 18:33 UTC (permalink / raw)
  To: clg; +Cc: famz, david, mst, qemu-devel, abologna, qemu-ppc

Hi,

This series failed build test on s390x host. Please find the details below.

Type: series
Message-id: 20180628083633.12413-1-clg@kaod.org
Subject: [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
echo "=== ENV ==="
env
echo "=== PACKAGES ==="
rpm -qa
echo "=== TEST BEGIN ==="
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
echo -n "Using CC: "
realpath $CC
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
e45f260ac0 ppc/pnv: make the PHB3 devices user creatable
f05ce5d541 ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge

=== OUTPUT BEGIN ===
=== ENV ===
LANG=en_US.UTF-8
XDG_SESSION_ID=253795
USER=fam
PWD=/var/tmp/patchew-tester-tmp-yhgce4ah/src
HOME=/home/fam
SHELL=/bin/sh
SHLVL=2
PATCHEW=/home/fam/patchew/patchew-cli -s http://patchew.org --nodebug
LOGNAME=fam
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1012/bus
XDG_RUNTIME_DIR=/run/user/1012
PATH=/usr/bin:/bin
_=/usr/bin/env
=== PACKAGES ===
gpg-pubkey-873529b8-54e386ff
glibc-debuginfo-common-2.24-10.fc25.s390x
fedora-release-26-1.noarch
dejavu-sans-mono-fonts-2.35-4.fc26.noarch
xemacs-filesystem-21.5.34-22.20170124hgf412e9f093d4.fc26.noarch
bash-4.4.12-7.fc26.s390x
libSM-1.2.2-5.fc26.s390x
libmpc-1.0.2-6.fc26.s390x
libaio-0.3.110-7.fc26.s390x
libverto-0.2.6-7.fc26.s390x
perl-Scalar-List-Utils-1.48-1.fc26.s390x
iptables-libs-1.6.1-2.fc26.s390x
tcl-8.6.6-2.fc26.s390x
libxshmfence-1.2-4.fc26.s390x
expect-5.45-23.fc26.s390x
perl-Thread-Queue-3.12-1.fc26.noarch
perl-encoding-2.19-6.fc26.s390x
keyutils-1.5.10-1.fc26.s390x
gmp-devel-6.1.2-4.fc26.s390x
enchant-1.6.0-16.fc26.s390x
python-gobject-base-3.24.1-1.fc26.s390x
python3-enchant-1.6.10-1.fc26.noarch
python-lockfile-0.11.0-6.fc26.noarch
python2-pyparsing-2.1.10-3.fc26.noarch
python2-lxml-4.1.1-1.fc26.s390x
librados2-10.2.7-2.fc26.s390x
trousers-lib-0.3.13-7.fc26.s390x
libdatrie-0.2.9-4.fc26.s390x
libsoup-2.58.2-1.fc26.s390x
passwd-0.79-9.fc26.s390x
bind99-libs-9.9.10-3.P3.fc26.s390x
python3-rpm-4.13.0.2-1.fc26.s390x
systemd-233-7.fc26.s390x
virglrenderer-0.6.0-1.20170210git76b3da97b.fc26.s390x
s390utils-ziomon-1.36.1-3.fc26.s390x
s390utils-osasnmpd-1.36.1-3.fc26.s390x
libXrandr-1.5.1-2.fc26.s390x
libglvnd-glx-1.0.0-1.fc26.s390x
texlive-ifxetex-svn19685.0.5-33.fc26.2.noarch
texlive-psnfss-svn33946.9.2a-33.fc26.2.noarch
texlive-dvipdfmx-def-svn40328-33.fc26.2.noarch
texlive-natbib-svn20668.8.31b-33.fc26.2.noarch
texlive-xdvi-bin-svn40750-33.20160520.fc26.2.s390x
texlive-cm-svn32865.0-33.fc26.2.noarch
texlive-beton-svn15878.0-33.fc26.2.noarch
texlive-fpl-svn15878.1.002-33.fc26.2.noarch
texlive-mflogo-svn38628-33.fc26.2.noarch
texlive-texlive-docindex-svn41430-33.fc26.2.noarch
texlive-luaotfload-bin-svn34647.0-33.20160520.fc26.2.noarch
texlive-koma-script-svn41508-33.fc26.2.noarch
texlive-pst-tree-svn24142.1.12-33.fc26.2.noarch
texlive-breqn-svn38099.0.98d-33.fc26.2.noarch
texlive-xetex-svn41438-33.fc26.2.noarch
gstreamer1-plugins-bad-free-1.12.3-1.fc26.s390x
xorg-x11-font-utils-7.5-33.fc26.s390x
ghostscript-fonts-5.50-36.fc26.noarch
libXext-devel-1.3.3-5.fc26.s390x
libusbx-devel-1.0.21-2.fc26.s390x
libglvnd-devel-1.0.0-1.fc26.s390x
emacs-25.3-3.fc26.s390x
alsa-lib-devel-1.1.4.1-1.fc26.s390x
kbd-2.0.4-2.fc26.s390x
dconf-0.26.0-2.fc26.s390x
mc-4.8.19-5.fc26.s390x
doxygen-1.8.13-9.fc26.s390x
dpkg-1.18.24-1.fc26.s390x
libtdb-1.3.13-1.fc26.s390x
python2-pynacl-1.1.1-1.fc26.s390x
perl-Filter-1.58-1.fc26.s390x
python2-pip-9.0.1-11.fc26.noarch
dnf-2.7.5-2.fc26.noarch
bind-license-9.11.2-1.P1.fc26.noarch
libtasn1-4.13-1.fc26.s390x
cpp-7.3.1-2.fc26.s390x
pkgconf-1.3.12-2.fc26.s390x
python2-fedora-0.10.0-1.fc26.noarch
cmake-filesystem-3.10.1-11.fc26.s390x
python3-requests-kerberos-0.12.0-1.fc26.noarch
libmicrohttpd-0.9.59-1.fc26.s390x
GeoIP-GeoLite-data-2018.01-1.fc26.noarch
python2-libs-2.7.14-7.fc26.s390x
libidn2-2.0.4-3.fc26.s390x
p11-kit-devel-0.23.10-1.fc26.s390x
perl-Errno-1.25-396.fc26.s390x
libdrm-2.4.90-2.fc26.s390x
sssd-common-1.16.1-1.fc26.s390x
boost-random-1.63.0-11.fc26.s390x
urw-fonts-2.4-24.fc26.noarch
ccache-3.3.6-1.fc26.s390x
glibc-debuginfo-2.24-10.fc25.s390x
dejavu-fonts-common-2.35-4.fc26.noarch
bind99-license-9.9.10-3.P3.fc26.noarch
ncurses-libs-6.0-8.20170212.fc26.s390x
libpng-1.6.28-2.fc26.s390x
libICE-1.0.9-9.fc26.s390x
perl-Text-ParseWords-3.30-366.fc26.noarch
libtool-ltdl-2.4.6-17.fc26.s390x
libselinux-utils-2.6-7.fc26.s390x
userspace-rcu-0.9.3-2.fc26.s390x
perl-Class-Inspector-1.31-3.fc26.noarch
keyutils-libs-devel-1.5.10-1.fc26.s390x
isl-0.16.1-1.fc26.s390x
libsecret-0.18.5-3.fc26.s390x
compat-openssl10-1.0.2m-1.fc26.s390x
python3-iniparse-0.4-24.fc26.noarch
python3-dateutil-2.6.0-3.fc26.noarch
python3-firewall-0.4.4.5-1.fc26.noarch
python-enum34-1.1.6-1.fc26.noarch
python2-pygments-2.2.0-7.fc26.noarch
python2-dockerfile-parse-0.0.7-1.fc26.noarch
perl-Net-SSLeay-1.81-1.fc26.s390x
hostname-3.18-2.fc26.s390x
libtirpc-1.0.2-0.fc26.s390x
rpm-build-libs-4.13.0.2-1.fc26.s390x
libutempter-1.1.6-9.fc26.s390x
systemd-pam-233-7.fc26.s390x
libXinerama-1.1.3-7.fc26.s390x
mesa-libGL-17.2.4-2.fc26.s390x
texlive-amsfonts-svn29208.3.04-33.fc26.2.noarch
texlive-caption-svn41409-33.fc26.2.noarch
texlive-enumitem-svn24146.3.5.2-33.fc26.2.noarch
texlive-pdftex-def-svn22653.0.06d-33.fc26.2.noarch
texlive-xdvi-svn40768-33.fc26.2.noarch
texlive-courier-svn35058.0-33.fc26.2.noarch
texlive-charter-svn15878.0-33.fc26.2.noarch
texlive-graphics-def-svn41879-33.fc26.2.noarch
texlive-mfnfss-svn19410.0-33.fc26.2.noarch
texlive-texlive-en-svn41185-33.fc26.2.noarch
texlive-ifplatform-svn21156.0.4-33.fc26.2.noarch
texlive-ms-svn29849.0-33.fc26.2.noarch
texlive-pst-tools-svn34067.0.05-33.fc26.2.noarch
texlive-powerdot-svn38984-33.fc26.2.noarch
texlive-xetexconfig-svn41133-33.fc26.2.noarch
libvdpau-1.1.1-4.fc26.s390x
zlib-devel-1.2.11-2.fc26.s390x
gdk-pixbuf2-devel-2.36.9-1.fc26.s390x
libX11-devel-1.6.5-2.fc26.s390x
libglvnd-core-devel-1.0.0-1.fc26.s390x
SDL2-devel-2.0.7-2.fc26.s390x
webkitgtk3-2.4.11-5.fc26.s390x
grubby-8.40-4.fc26.s390x
uboot-tools-2017.05-4.fc26.s390x
cracklib-dicts-2.9.6-5.fc26.s390x
texinfo-6.3-3.fc26.s390x
time-1.7-52.fc26.s390x
python2-deltarpm-3.6-19.fc26.s390x
python2-setuptools-37.0.0-1.fc26.noarch
python2-dnf-2.7.5-2.fc26.noarch
groff-base-1.22.3-10.fc26.s390x
python2-GitPython-2.1.7-2.fc26.noarch
cups-libs-2.2.2-8.fc26.s390x
bind-libs-lite-9.11.2-1.P1.fc26.s390x
libpkgconf-1.3.12-2.fc26.s390x
java-1.8.0-openjdk-headless-1.8.0.161-5.b14.fc26.s390x
python3-dnf-plugin-system-upgrade-2.0.5-1.fc26.noarch
dtc-1.4.6-1.fc26.s390x
glusterfs-client-xlators-3.10.11-1.fc26.s390x
libunistring-0.9.9-1.fc26.s390x
python3-libs-3.6.4-3.fc26.s390x
perl-IO-1.36-396.fc26.s390x
libXcursor-1.1.15-1.fc26.s390x
libdrm-devel-2.4.90-2.fc26.s390x
boost-thread-1.63.0-11.fc26.s390x
strace-4.21-1.fc26.s390x
boost-iostreams-1.63.0-11.fc26.s390x
gpg-pubkey-efe550f5-5220ba41
gpg-pubkey-81b46521-55b3ca9a
filesystem-3.2-40.fc26.s390x
basesystem-11-3.fc26.noarch
js-jquery-3.2.1-1.fc26.noarch
libidn-1.33-2.fc26.s390x
libogg-1.3.2-6.fc26.s390x
slang-2.3.1a-2.fc26.s390x
apr-1.6.3-1.fc26.s390x
libxkbcommon-0.7.1-3.fc26.s390x
less-487-3.fc26.s390x
lttng-ust-2.9.0-2.fc26.s390x
OpenEXR-libs-2.2.0-6.fc26.s390x
ipset-libs-6.29-3.fc26.s390x
perl-XML-XPath-1.42-1.fc26.noarch
lua-filesystem-1.6.3-3.fc24.s390x
gstreamer1-1.12.3-1.fc26.s390x
libpwquality-1.3.0-8.fc26.s390x
gettext-libs-0.19.8.1-9.fc26.s390x
python3-chardet-2.3.0-3.fc26.noarch
python3-slip-dbus-0.6.4-6.fc26.noarch
python-chardet-2.3.0-3.fc26.noarch
python2-pyasn1-0.2.3-1.fc26.noarch
python-slip-dbus-0.6.4-6.fc26.noarch
libarchive-3.2.2-4.fc26.s390x
libbabeltrace-1.5.2-2.fc26.s390x
cdparanoia-libs-10.2-22.fc26.s390x
gpgme-1.8.0-12.fc26.s390x
python2-gpg-1.8.0-12.fc26.s390x
shadow-utils-4.3.1-3.fc26.s390x
cryptsetup-libs-1.7.5-1.fc26.s390x
kpartx-0.4.9-88.fc26.s390x
libXi-1.7.9-2.fc26.s390x
texlive-tetex-svn41059-33.fc26.2.noarch
texlive-tools-svn40934-33.fc26.2.noarch
texlive-bibtex-bin-svn40473-33.20160520.fc26.2.s390x
texlive-mfware-bin-svn40473-33.20160520.fc26.2.s390x
texlive-underscore-svn18261.0-33.fc26.2.noarch
texlive-avantgar-svn31835.0-33.fc26.2.noarch
texlive-anysize-svn15878.0-33.fc26.2.noarch
texlive-lineno-svn21442.4.41-33.fc26.2.noarch
texlive-mathpazo-svn15878.1.003-33.fc26.2.noarch
texlive-soul-svn15878.2.4-33.fc26.2.noarch
texlive-luatexbase-svn38550-33.fc26.2.noarch
texlive-listings-svn37534.1.6-33.fc26.2.noarch
texlive-pstricks-svn41321-33.fc26.2.noarch
texlive-metalogo-svn18611.0.12-33.fc26.2.noarch
texlive-dvipdfmx-svn41149-33.fc26.2.noarch
kbd-legacy-2.0.4-2.fc26.noarch
ghostscript-x11-9.20-10.fc26.s390x
libXrender-devel-0.9.10-2.fc26.s390x
libxkbcommon-devel-0.7.1-3.fc26.s390x
mesa-libGL-devel-17.2.4-2.fc26.s390x
usbredir-devel-0.7.1-3.fc26.s390x
libcap-devel-2.25-5.fc26.s390x
brlapi-devel-0.6.6-5.fc26.s390x
python3-pygpgme-0.3-22.fc26.s390x
pinentry-0.9.7-3.fc26.s390x
qemu-sanity-check-nodeps-1.1.5-6.fc26.s390x
libldb-1.1.29-5.fc26.s390x
libwayland-cursor-1.13.0-3.fc26.s390x
pulseaudio-libs-devel-11.1-7.fc26.s390x
json-c-0.12.1-5.fc26.s390x
libgcrypt-1.8.2-1.fc26.s390x
libgo-devel-7.3.1-2.fc26.s390x
ca-certificates-2018.2.22-1.0.fc26.noarch
python2-sphinx-1.5.6-1.fc26.noarch
dnsmasq-2.76-6.fc26.s390x
perl-Module-CoreList-5.20180120-1.fc26.noarch
pcre-8.41-6.fc26.s390x
net-snmp-libs-5.7.3-27.fc26.s390x
gnutls-dane-3.5.18-2.fc26.s390x
glusterfs-devel-3.10.11-1.fc26.s390x
libsss_nss_idmap-1.16.1-1.fc26.s390x
elfutils-0.170-4.fc26.s390x
nss-devel-3.36.0-1.0.fc26.s390x
perl-open-1.10-396.fc26.noarch
ethtool-4.15-1.fc26.s390x
gpg-pubkey-34ec9cba-54e38751
gpg-pubkey-030d5aed-55b577f0
setup-2.10.5-2.fc26.noarch
lato-fonts-2.015-3.fc26.noarch
web-assets-filesystem-5-5.fc26.noarch
libsepol-2.6-2.fc26.s390x
libcap-2.25-5.fc26.s390x
tcp_wrappers-libs-7.6-85.fc26.s390x
libnl3-3.3.0-1.fc26.s390x
pixman-0.34.0-3.fc26.s390x
lzo-2.08-9.fc26.s390x
libnl3-cli-3.3.0-1.fc26.s390x
gpm-libs-1.20.7-10.fc26.s390x
iso-codes-3.74-2.fc26.noarch
ipset-6.29-3.fc26.s390x
lua-term-0.07-1.fc25.s390x
libdb-utils-5.3.28-24.fc26.s390x
dbus-glib-0.108-2.fc26.s390x
pam-1.3.0-2.fc26.s390x
avahi-glib-0.6.32-7.fc26.s390x
python2-dateutil-2.6.0-3.fc26.noarch
python3-asn1crypto-0.23.0-1.fc26.noarch
python3-slip-0.6.4-6.fc26.noarch
python-backports-ssl_match_hostname-3.5.0.1-4.fc26.noarch
python2-pyOpenSSL-16.2.0-6.fc26.noarch
python-slip-0.6.4-6.fc26.noarch
nss-pem-1.0.3-3.fc26.s390x
fipscheck-1.5.0-1.fc26.s390x
cyrus-sasl-lib-2.1.26-32.fc26.s390x
python3-kerberos-1.2.5-3.fc26.s390x
rpmconf-1.0.19-1.fc26.noarch
libsemanage-2.6-4.fc26.s390x
device-mapper-libs-1.02.137-6.fc26.s390x
yum-3.4.3-512.fc26.noarch
device-mapper-multipath-0.4.9-88.fc26.s390x
libXtst-1.2.3-2.fc26.s390x
libXxf86vm-1.1.4-4.fc26.s390x
texlive-amsmath-svn41561-33.fc26.2.noarch
texlive-xkeyval-svn35741.2.7a-33.fc26.2.noarch
texlive-bibtex-svn40768-33.fc26.2.noarch
texlive-mfware-svn40768-33.fc26.2.noarch
texlive-wasy-svn35831.0-33.fc26.2.noarch
texlive-bookman-svn31835.0-33.fc26.2.noarch
texlive-babel-english-svn30264.3.3p-33.fc26.2.noarch
texlive-fix2col-svn38770-33.fc26.2.noarch
texlive-mdwtools-svn15878.1.05.4-33.fc26.2.noarch
texlive-tex-gyre-math-svn41264-33.fc26.2.noarch
texlive-luaotfload-svn40902-33.fc26.2.noarch
texlive-showexpl-svn32737.v0.3l-33.fc26.2.noarch
texlive-pstricks-add-svn40744-33.fc26.2.noarch
texlive-l3experimental-svn41163-33.fc26.2.noarch
texlive-xetex-bin-svn41091-33.20160520.fc26.2.s390x
kbd-misc-2.0.4-2.fc26.noarch
libpng-devel-1.6.28-2.fc26.s390x
ghostscript-core-9.20-10.fc26.s390x
libXfixes-devel-5.0.3-2.fc26.s390x
libverto-devel-0.2.6-7.fc26.s390x
mesa-libEGL-devel-17.2.4-2.fc26.s390x
popt-devel-1.16-12.fc26.s390x
readline-devel-7.0-5.fc26.s390x
cyrus-sasl-devel-2.1.26-32.fc26.s390x
sendmail-8.15.2-19.fc26.s390x
systemd-bootchart-231-3.fc26.s390x
perl-IO-Socket-SSL-2.049-1.fc26.noarch
python2-enchant-1.6.10-1.fc26.noarch
perl-generators-1.10-2.fc26.noarch
createrepo-0.10.3-11.fc26.noarch
pulseaudio-libs-glib2-11.1-7.fc26.s390x
dhcp-libs-4.3.5-10.fc26.s390x
libtiff-4.0.9-1.fc26.s390x
python-srpm-macros-3-21.fc26.noarch
libtalloc-2.1.11-1.fc26.s390x
nfs-utils-2.2.1-4.rc2.fc26.s390x
qt5-srpm-macros-5.9.4-2.fc26.noarch
python2-dnf-plugins-core-2.1.5-4.fc26.noarch
mariadb-libs-10.1.30-2.fc26.s390x
bind-libs-9.11.2-1.P1.fc26.s390x
acpica-tools-20180105-1.fc26.s390x
perl-podlators-4.09-3.fc26.noarch
glusterfs-3.10.11-1.fc26.s390x
nss-sysinit-3.36.0-1.0.fc26.s390x
gnutls-c++-3.5.18-2.fc26.s390x
perl-macros-5.24.3-396.fc26.s390x
sssd-client-1.16.1-1.fc26.s390x
elfutils-devel-0.170-4.fc26.s390x
kernel-4.15.12-201.fc26.s390x
vim-minimal-8.0.1553-1.fc26.s390x
desktop-file-utils-0.23-6.fc26.s390x
fontpackages-filesystem-1.44-18.fc26.noarch
vte-profile-0.48.4-1.fc26.s390x
texlive-kpathsea-doc-svn41139-33.fc26.2.noarch
zlib-1.2.11-2.fc26.s390x
readline-7.0-5.fc26.s390x
libattr-2.4.47-18.fc26.s390x
libglvnd-1.0.0-1.fc26.s390x
lz4-libs-1.8.0-1.fc26.s390x
perl-File-Path-2.12-367.fc26.noarch
perl-Unicode-EastAsianWidth-1.33-9.fc26.noarch
hunspell-1.5.4-2.fc26.s390x
libasyncns-0.8-11.fc26.s390x
libnetfilter_conntrack-1.0.6-2.fc26.s390x
perl-Storable-2.56-368.fc26.s390x
autoconf-2.69-24.fc26.noarch
device-mapper-persistent-data-0.6.3-5.fc26.s390x
quota-4.03-9.fc26.s390x
crypto-policies-20170606-1.git7c32281.fc26.noarch
glib2-2.52.3-2.fc26.s390x
python2-idna-2.5-1.fc26.noarch
python2-libcomps-0.1.8-3.fc26.s390x
gsettings-desktop-schemas-3.24.1-1.fc26.s390x
javapackages-tools-4.7.0-17.fc26.noarch
libselinux-python3-2.6-7.fc26.s390x
python-backports-1.0-9.fc26.s390x
python2-cryptography-2.0.2-2.fc26.s390x
libselinux-python-2.6-7.fc26.s390x
Lmod-7.5.3-1.fc26.s390x
fipscheck-lib-1.5.0-1.fc26.s390x
libuser-0.62-6.fc26.s390x
npth-1.5-1.fc26.s390x
packagedb-cli-2.14.1-2.fc26.noarch
ustr-1.0.4-22.fc26.s390x
device-mapper-1.02.137-6.fc26.s390x
polkit-pkla-compat-0.1-8.fc26.s390x
fakeroot-1.22-1.fc26.s390x
libXmu-1.1.2-5.fc26.s390x
cairo-gobject-1.14.10-1.fc26.s390x
texlive-booktabs-svn40846-33.fc26.2.noarch
texlive-dvips-bin-svn40987-33.20160520.fc26.2.s390x
texlive-float-svn15878.1.3d-33.fc26.2.noarch
texlive-tex-svn40793-33.fc26.2.noarch
texlive-fancyref-svn15878.0.9c-33.fc26.2.noarch
texlive-manfnt-font-svn35799.0-33.fc26.2.noarch
texlive-cmap-svn41168-33.fc26.2.noarch
texlive-hyph-utf8-svn41189-33.fc26.2.noarch
texlive-paralist-svn39247-33.fc26.2.noarch
texlive-trimspaces-svn15878.1.1-33.fc26.2.noarch
texlive-tipa-svn29349.1.3-33.fc26.2.noarch
texlive-l3packages-svn41246-33.fc26.2.noarch
texlive-pst-pdf-svn31660.1.1v-33.fc26.2.noarch
texlive-tex-gyre-svn18651.2.004-33.fc26.2.noarch
texlive-beamer-svn36461.3.36-33.fc26.2.noarch
gd-2.2.5-1.fc26.s390x
gc-devel-7.6.0-2.fc26.s390x
libXft-devel-2.3.2-5.fc26.s390x
rpm-devel-4.13.0.2-1.fc26.s390x
bluez-libs-devel-5.46-6.fc26.s390x
trousers-0.3.13-7.fc26.s390x
iproute-tc-4.11.0-1.fc26.s390x
libgnome-keyring-3.12.0-8.fc26.s390x
perl-File-ShareDir-1.102-8.fc26.noarch
python2-paramiko-2.2.1-1.fc26.noarch
python2-openidc-client-0.4.0-1.20171113git54dee6e.fc26.noarch
openssh-server-7.5p1-4.fc26.s390x
pulseaudio-libs-11.1-7.fc26.s390x
python2-bodhi-2.12.2-3.fc26.noarch
lua-libs-5.3.4-7.fc26.s390x
dhcp-common-4.3.5-10.fc26.noarch
python3-pip-9.0.1-11.fc26.noarch
python2-py-1.4.34-1.fc26.noarch
glibc-common-2.25-13.fc26.s390x
webkitgtk4-jsc-2.18.6-1.fc26.s390x
glibc-devel-2.25-13.fc26.s390x
pcre2-10.23-13.fc26.s390x
linux-firmware-20171215-82.git2451bb22.fc26.noarch
libfdt-devel-1.4.6-1.fc26.s390x
audit-2.8.2-1.fc26.s390x
perl-Socket-2.027-1.fc26.s390x
nosync-1.0-6.fc26.s390x
redhat-rpm-config-65-1.fc26.noarch
freetype-2.7.1-10.fc26.s390x
gnutls-3.5.18-2.fc26.s390x
sqlite-3.20.1-2.fc26.s390x
pcre-devel-8.41-6.fc26.s390x
fedpkg-1.32-1.fc26.noarch
gnutls-devel-3.5.18-2.fc26.s390x
python2-pytz-2017.2-7.fc26.noarch
gsm-1.0.17-2.fc26.s390x
gpg-pubkey-95a43f54-5284415a
gpg-pubkey-fdb19c98-56fd6333
gpg-pubkey-64dab85d-57d33e22
firewalld-filesystem-0.4.4.5-1.fc26.noarch
xkeyboard-config-2.21-3.fc26.noarch
texlive-texlive-common-doc-svn40682-33.fc26.2.noarch
ncurses-base-6.0-8.20170212.fc26.noarch
libselinux-2.6-7.fc26.s390x
bzip2-libs-1.0.6-22.fc26.s390x
libdb-5.3.28-24.fc26.s390x
file-libs-5.30-11.fc26.s390x
libxslt-1.1.29-1.fc26.s390x
gdbm-1.13-1.fc26.s390x
libepoxy-1.4.3-1.fc26.s390x
libpsl-0.18.0-1.fc26.s390x
perl-Carp-1.40-366.fc26.noarch
e2fsprogs-libs-1.43.4-2.fc26.s390x
libmnl-1.0.4-2.fc26.s390x
openjpeg2-2.2.0-3.fc26.s390x
perl-PathTools-3.63-367.fc26.s390x
perl-File-Temp-0.230.400-2.fc26.noarch
perl-XML-Parser-2.44-6.fc26.s390x
libss-1.43.4-2.fc26.s390x
ilmbase-2.2.0-8.fc26.s390x
fuse-libs-2.9.7-2.fc26.s390x
libdaemon-0.14-11.fc26.s390x
libbasicobjects-0.1.1-34.fc26.s390x
iptables-1.6.1-2.fc26.s390x
perl-TermReadKey-2.37-2.fc26.s390x
perl-Term-ANSIColor-4.06-2.fc26.noarch
perl-libintl-perl-1.26-2.fc26.s390x
usbredir-0.7.1-3.fc26.s390x
fftw-libs-double-3.3.5-4.fc26.s390x
libiscsi-1.15.0-3.fc26.s390x
ttmkfdir-3.0.9-49.fc26.s390x
texlive-base-2016-33.20160520.fc26.1.noarch
python2-six-1.10.0-9.fc26.noarch
atk-2.24.0-1.fc26.s390x
python2-kitchen-1.2.4-6.fc26.noarch
guile-2.0.14-1.fc26.s390x
pyxattr-0.5.3-10.fc26.s390x
libyaml-0.1.7-2.fc26.s390x
python3-PyYAML-3.12-3.fc26.s390x
openssh-7.5p1-4.fc26.s390x
openssl-1.1.0g-1.fc26.s390x
gawk-4.1.4-6.fc26.s390x
openldap-2.4.45-2.fc26.s390x
NetworkManager-libnm-1.8.2-4.fc26.s390x
python2-urllib3-1.20-2.fc26.noarch
python3-py-1.4.34-1.fc26.noarch
perl-ExtUtils-Command-7.24-3.fc26.noarch
tzdata-2018c-1.fc26.noarch
libcrypt-nss-2.25-13.fc26.s390x
libstdc++-devel-7.3.1-2.fc26.s390x
rpcbind-0.2.4-8.rc3.fc26.s390x
gdb-headless-8.0.1-36.fc26.s390x
python3-dnf-plugins-extras-common-2.0.5-1.fc26.noarch
glibc-headers-2.25-13.fc26.s390x
libfdt-1.4.6-1.fc26.s390x
wget-1.19.4-1.fc26.s390x
mariadb-common-10.1.30-2.fc26.s390x
python2-dnf-plugin-migrate-2.1.5-4.fc26.noarch
pcre2-devel-10.23-13.fc26.s390x
perl-threads-shared-1.58-1.fc26.s390x
gcc-c++-7.3.1-2.fc26.s390x
ImageMagick-libs-6.9.9.27-1.fc26.s390x
poppler-0.52.0-11.fc26.s390x
perl-Data-Dumper-2.161-4.fc26.s390x
python2-dnf-plugins-extras-common-2.0.5-1.fc26.noarch
gcc-debuginfo-7.3.1-2.fc26.s390x
krb5-libs-1.15.2-7.fc26.s390x
nspr-devel-4.19.0-1.fc26.s390x
nss-softokn-3.36.0-1.0.fc26.s390x
libsss_idmap-1.16.1-1.fc26.s390x
systemtap-runtime-3.2-7.fc26.s390x
gnupg2-2.2.5-1.fc26.s390x
python2-gluster-3.10.11-1.fc26.s390x
sqlite-devel-3.20.1-2.fc26.s390x
git-2.13.6-3.fc26.s390x
libtevent-0.9.36-1.fc26.s390x
elfutils-libs-0.170-4.fc26.s390x
systemtap-3.2-7.fc26.s390x
vim-enhanced-8.0.1553-1.fc26.s390x
gnupg2-smime-2.2.5-1.fc26.s390x
libcurl-devel-7.53.1-16.fc26.s390x
python2-sssdconfig-1.16.1-1.fc26.noarch
patch-2.7.6-3.fc26.s390x
fedora-repos-26-3.noarch
python3-mock-2.0.0-4.fc26.noarch
libgudev-232-1.fc26.s390x
python3-javapackages-4.7.0-17.fc26.noarch
python3-ply-3.9-3.fc26.noarch
python3-systemd-234-1.fc26.s390x
python3-requests-2.13.0-1.fc26.noarch
blktrace-1.1.0-4.fc26.s390x
python2-asn1crypto-0.23.0-1.fc26.noarch
python2-cffi-1.9.1-2.fc26.s390x
python2-sphinx_rtd_theme-0.2.4-1.fc26.noarch
lua-json-1.3.2-7.fc26.noarch
libcephfs1-10.2.7-2.fc26.s390x
glib-networking-2.50.0-2.fc26.s390x
libedit-3.1-17.20160618cvs.fc26.s390x
libverto-libev-0.2.6-7.fc26.s390x
libserf-1.3.9-3.fc26.s390x
python2-kerberos-1.2.5-3.fc26.s390x
libsrtp-1.5.4-4.fc26.s390x
lzo-minilzo-2.08-9.fc26.s390x
librepo-1.8.0-1.fc26.s390x
sg3_utils-1.42-1.fc26.s390x
policycoreutils-2.6-6.fc26.s390x
lvm2-2.02.168-6.fc26.s390x
device-mapper-multipath-libs-0.4.9-88.fc26.s390x
s390utils-cmsfs-1.36.1-3.fc26.s390x
libXdamage-1.1.4-9.fc26.s390x
libXaw-1.0.13-5.fc26.s390x
brltty-5.5-5.fc26.s390x
librsvg2-2.40.18-1.fc26.s390x
texlive-tetex-bin-svn36770.0-33.20160520.fc26.2.noarch
texlive-etex-pkg-svn39355-33.fc26.2.noarch
texlive-graphics-svn41015-33.fc26.2.noarch
texlive-dvips-svn41149-33.fc26.2.noarch
texlive-zapfding-svn31835.0-33.fc26.2.noarch
texlive-footmisc-svn23330.5.5b-33.fc26.2.noarch
texlive-makeindex-svn40768-33.fc26.2.noarch
texlive-pst-ovl-svn40873-33.fc26.2.noarch
texlive-texlive-scripts-svn41433-33.fc26.2.noarch
texlive-ltabptch-svn17533.1.74d-33.fc26.2.noarch
texlive-euro-svn22191.1.1-33.fc26.2.noarch
texlive-mflogo-font-svn36898.1.002-33.fc26.2.noarch
texlive-zapfchan-svn31835.0-33.fc26.2.noarch
texlive-cmextra-svn32831.0-33.fc26.2.noarch
texlive-finstrut-svn21719.0.5-33.fc26.2.noarch
texlive-hyphen-base-svn41138-33.fc26.2.noarch
texlive-marginnote-svn41382-33.fc26.2.noarch
texlive-parallel-svn15878.0-33.fc26.2.noarch
texlive-sepnum-svn20186.2.0-33.fc26.2.noarch
texlive-environ-svn33821.0.3-33.fc26.2.noarch
texlive-type1cm-svn21820.0-33.fc26.2.noarch
texlive-xunicode-svn30466.0.981-33.fc26.2.noarch
texlive-attachfile-svn38830-33.fc26.2.noarch
texlive-fontspec-svn41262-33.fc26.2.noarch
texlive-fancyvrb-svn18492.2.8-33.fc26.2.noarch
texlive-pst-pdf-bin-svn7838.0-33.20160520.fc26.2.noarch
texlive-xcolor-svn41044-33.fc26.2.noarch
texlive-pdfpages-svn40638-33.fc26.2.noarch
texlive-sansmathaccent-svn30187.0-33.fc26.2.noarch
texlive-ucs-svn35853.2.2-33.fc26.2.noarch
texlive-dvipdfmx-bin-svn40273-33.20160520.fc26.2.s390x
libotf-0.9.13-8.fc26.s390x
go-srpm-macros-2-8.fc26.noarch
mesa-libwayland-egl-devel-17.2.4-2.fc26.s390x
ghostscript-9.20-10.fc26.s390x
libcephfs_jni-devel-10.2.7-2.fc26.s390x
libXdamage-devel-1.1.4-9.fc26.s390x
ncurses-devel-6.0-8.20170212.fc26.s390x
fontconfig-devel-2.12.6-4.fc26.s390x
cairo-devel-1.14.10-1.fc26.s390x
libselinux-devel-2.6-7.fc26.s390x
guile-devel-2.0.14-1.fc26.s390x
libcap-ng-devel-0.7.8-3.fc26.s390x
bash-completion-2.6-1.fc26.noarch
libXevie-1.0.3-12.fc26.s390x
python-firewall-0.4.4.5-1.fc26.noarch
python3-html5lib-0.999-13.fc26.noarch
python2-simplejson-3.10.0-3.fc26.s390x
flex-2.6.1-3.fc26.s390x
telnet-0.17-69.fc26.s390x
gpg-pubkey-8e1431d5-53bcbac7
emacs-filesystem-25.3-3.fc26.noarch
fontawesome-fonts-4.7.0-2.fc26.noarch
fontawesome-fonts-web-4.7.0-2.fc26.noarch
rpmconf-base-1.0.19-1.fc26.noarch
info-6.3-3.fc26.s390x
texlive-lib-2016-33.20160520.fc26.1.s390x
libicu-57.1-7.fc26.s390x
libcap-ng-0.7.8-3.fc26.s390x
nettle-3.3-2.fc26.s390x
lcms2-2.8-3.fc26.s390x
dbus-libs-1.11.18-1.fc26.s390x
perl-Exporter-5.72-367.fc26.noarch
unzip-6.0-34.fc26.s390x
iproute-4.11.0-1.fc26.s390x
zip-3.0-18.fc26.s390x
perl-constant-1.33-368.fc26.noarch
perl-MIME-Base64-3.15-366.fc26.s390x
lua-posix-33.3.1-4.fc26.s390x
bzip2-1.0.6-22.fc26.s390x
hyphen-2.8.8-6.fc26.s390x
libdvdread-5.0.3-4.fc26.s390x
libcollection-0.7.0-34.fc26.s390x
libdvdnav-5.0.3-5.fc26.s390x
perl-version-0.99.18-1.fc26.s390x
perl-Encode-2.88-6.fc26.s390x
automake-1.15-9.fc26.noarch
plymouth-core-libs-0.9.3-0.7.20160620git0e65b86c.fc26.s390x
hesiod-3.2.1-7.fc26.s390x
jasper-libs-2.0.14-1.fc26.s390x
mozjs17-17.0.0-18.fc26.s390x
fontconfig-2.12.6-4.fc26.s390x
harfbuzz-1.4.4-1.fc26.s390x
alsa-lib-1.1.4.1-1.fc26.s390x
make-4.2.1-2.fc26.s390x
gobject-introspection-1.52.1-1.fc26.s390x
hicolor-icon-theme-0.15-5.fc26.noarch
gdk-pixbuf2-2.36.9-1.fc26.s390x
libgusb-0.2.11-1.fc26.s390x
libdhash-0.5.0-34.fc26.s390x
python2-bcrypt-3.1.4-2.fc26.s390x
PyYAML-3.12-3.fc26.s390x
openssl-devel-1.1.0g-1.fc26.s390x
copy-jdk-configs-3.3-2.fc26.noarch
python3-setuptools-37.0.0-1.fc26.noarch
kernel-core-4.14.8-200.fc26.s390x
NetworkManager-1.8.2-4.fc26.s390x
libjpeg-turbo-devel-1.5.3-1.fc26.s390x
lua-5.3.4-7.fc26.s390x
kernel-devel-4.14.8-200.fc26.s390x
perl-autodie-2.29-367.fc26.noarch
tzdata-java-2018c-1.fc26.noarch
createrepo_c-0.10.0-15.fc26.s390x
libgfortran-7.3.1-2.fc26.s390x
mariadb-config-10.1.30-2.fc26.s390x
java-1.8.0-openjdk-1.8.0.161-5.b14.fc26.s390x
libtasn1-devel-4.13-1.fc26.s390x
gcc-gdb-plugin-7.3.1-2.fc26.s390x
python2-libxml2-2.9.7-1.fc26.s390x
net-tools-2.0-0.44.20160912git.fc26.s390x
python2-requests-kerberos-0.12.0-1.fc26.noarch
gcc-base-debuginfo-7.3.1-2.fc26.s390x
glusterfs-libs-3.10.11-1.fc26.s390x
system-python-libs-3.6.4-3.fc26.s390x
nss-softokn-freebl-3.36.0-1.0.fc26.s390x
git-core-2.13.6-3.fc26.s390x
libsss_certmap-1.16.1-1.fc26.s390x
nss-softokn-devel-3.36.0-1.0.fc26.s390x
python3-3.6.4-3.fc26.s390x
glusterfs-cli-3.10.11-1.fc26.s390x
perl-5.24.3-396.fc26.s390x
pcre-utf32-8.41-6.fc26.s390x
kernel-headers-4.15.12-201.fc26.s390x
mock-1.4.9-1.fc26.noarch
libXcursor-devel-1.1.15-1.fc26.s390x
python3-sssdconfig-1.16.1-1.fc26.noarch
freetype-devel-2.7.1-10.fc26.s390x
python2-devel-2.7.14-7.fc26.s390x
sssd-nfs-idmap-1.16.1-1.fc26.s390x
libsss_autofs-1.16.1-1.fc26.s390x
libzip-1.3.0-1.fc26.s390x
python3-lxml-4.1.1-1.fc26.s390x
python3-ordered-set-2.0.0-6.fc26.noarch
python3-rpmconf-1.0.19-1.fc26.noarch
python-offtrac-0.1.0-9.fc26.noarch
python2-pycparser-2.14-10.fc26.noarch
python2-sphinx-theme-alabaster-0.7.9-3.fc26.noarch
python2-pysocks-1.6.7-1.fc26.noarch
lua-lpeg-1.0.1-2.fc26.s390x
libproxy-0.4.15-2.fc26.s390x
crontabs-1.11-14.20150630git.fc26.noarch
libev-4.24-2.fc26.s390x
libsigsegv-2.11-1.fc26.s390x
fedora-cert-0.6.0.1-2.fc26.noarch
drpm-0.3.0-6.fc26.s390x
python2-cccolutils-1.5-3.fc26.s390x
m17n-lib-1.7.0-6.fc26.s390x
lsscsi-0.28-4.fc26.s390x
python3-gpg-1.8.0-12.fc26.s390x
sg3_utils-libs-1.42-1.fc26.s390x
SDL2-2.0.7-2.fc26.s390x
util-linux-2.30.2-1.fc26.s390x
s390utils-mon_statd-1.36.1-3.fc26.s390x
GConf2-3.2.6-17.fc26.s390x
systemd-container-233-7.fc26.s390x
libXt-1.1.5-4.fc26.s390x
libXpm-3.5.12-2.fc26.s390x
at-spi2-core-2.24.1-1.fc26.s390x
cairo-1.14.10-1.fc26.s390x
texlive-kpathsea-bin-svn40473-33.20160520.fc26.2.s390x
texlive-ifluatex-svn41346-33.fc26.2.noarch
texlive-babel-svn40706-33.fc26.2.noarch
texlive-colortbl-svn29803.v1.0a-33.fc26.2.noarch
texlive-marvosym-svn29349.2.2a-33.fc26.2.noarch
texlive-euler-svn17261.2.5-33.fc26.2.noarch
texlive-latexconfig-svn40274-33.fc26.2.noarch
texlive-plain-svn40274-33.fc26.2.noarch
texlive-texconfig-bin-svn29741.0-33.20160520.fc26.2.noarch
giflib-4.1.6-16.fc26.s390x
texlive-microtype-svn41127-33.fc26.2.noarch
texlive-eurosym-svn17265.1.4_subrfix-33.fc26.2.noarch
texlive-symbol-svn31835.0-33.fc26.2.noarch
texlive-chngcntr-svn17157.1.0a-33.fc26.2.noarch
texlive-euenc-svn19795.0.1h-33.fc26.2.noarch
texlive-luatex-svn40963-33.fc26.2.noarch
texlive-knuth-local-svn38627-33.fc26.2.noarch
texlive-mparhack-svn15878.1.4-33.fc26.2.noarch
texlive-rcs-svn15878.0-33.fc26.2.noarch
texlive-texlive-msg-translations-svn41431-33.fc26.2.noarch
texlive-updmap-map-svn41159-33.fc26.2.noarch
texlive-geometry-svn19716.5.6-33.fc26.2.noarch
texlive-memoir-svn41203-33.fc26.2.noarch
texlive-l3kernel-svn41246-33.fc26.2.noarch
texlive-pst-eps-svn15878.1.0-33.fc26.2.noarch
texlive-pst-text-svn15878.1.00-33.fc26.2.noarch
texlive-amscls-svn36804.0-33.fc26.2.noarch
texlive-pst-slpe-svn24391.1.31-33.fc26.2.noarch
texlive-extsizes-svn17263.1.4a-33.fc26.2.noarch
texlive-xetex-def-svn40327-33.fc26.2.noarch
texlive-collection-latex-svn41011-33.20160520.fc26.2.noarch
gstreamer1-plugins-base-1.12.3-1.fc26.s390x
fpc-srpm-macros-1.1-2.fc26.noarch
xorg-x11-proto-devel-7.7-22.fc26.noarch
atk-devel-2.24.0-1.fc26.s390x
libxcb-devel-1.12-3.fc26.s390x
libXrandr-devel-1.5.1-2.fc26.s390x
libcom_err-devel-1.43.4-2.fc26.s390x
dbus-devel-1.11.18-1.fc26.s390x
libepoxy-devel-1.4.3-1.fc26.s390x
libicu-devel-57.1-7.fc26.s390x
rpm-build-4.13.0.2-1.fc26.s390x
libssh2-devel-1.8.0-5.fc26.s390x
graphviz-2.40.1-4.fc26.s390x
zlib-static-1.2.11-2.fc26.s390x
mesa-libgbm-devel-17.2.4-2.fc26.s390x
screen-4.6.2-1.fc26.s390x
python-osbs-client-0.39.1-1.fc26.noarch
pyparsing-2.1.10-3.fc26.noarch
python3-pyasn1-0.2.3-1.fc26.noarch
python2-html5lib-0.999-13.fc26.noarch
teamd-1.27-1.fc26.s390x
hardlink-1.3-1.fc26.s390x
chrpath-0.16-4.fc26.s390x
texlive-pdftex-doc-svn41149-33.fc26.2.noarch
grep-3.1-1.fc26.s390x
libacl-2.2.52-15.fc26.s390x
cpio-2.12-4.fc26.s390x
libatomic_ops-7.4.4-2.fc26.s390x
gc-7.6.0-2.fc26.s390x
psmisc-22.21-9.fc26.s390x
systemd-libs-233-7.fc26.s390x
xz-5.2.3-2.fc26.s390x
libpcap-1.8.1-3.fc26.s390x
perl-parent-0.236-2.fc26.noarch
perl-Text-Unidecode-1.30-2.fc26.noarch
newt-0.52.20-1.fc26.s390x
libcomps-0.1.8-3.fc26.s390x
libfontenc-1.1.3-4.fc26.s390x
ipcalc-0.2.0-1.fc26.s390x
libnfnetlink-1.0.1-9.fc26.s390x
libref_array-0.1.5-34.fc26.s390x
perl-Term-Cap-1.17-366.fc26.noarch
perl-Digest-1.17-367.fc26.noarch
perl-Pod-Simple-3.35-2.fc26.noarch
perl-URI-1.71-6.fc26.noarch
attr-2.4.47-18.fc26.s390x
gmp-c++-6.1.2-4.fc26.s390x
harfbuzz-icu-1.4.4-1.fc26.s390x
http-parser-2.7.1-5.fc26.s390x
libsodium-1.0.14-1.fc26.s390x
python-gssapi-1.2.0-5.fc26.s390x
perl-libnet-3.11-1.fc26.noarch
libwayland-client-1.13.0-3.fc26.s390x
python3-dnf-2.7.5-2.fc26.noarch
kernel-modules-4.14.8-200.fc26.s390x
NetworkManager-ppp-1.8.2-4.fc26.s390x
wayland-devel-1.13.0-3.fc26.s390x
kernel-4.14.8-200.fc26.s390x
NetworkManager-glib-1.8.2-4.fc26.s390x
perl-IPC-System-Simple-1.25-12.fc26.noarch
sed-4.4-2.fc26.s390x
libassuan-2.5.1-1.fc26.s390x
createrepo_c-libs-0.10.0-15.fc26.s390x
dnf-utils-2.1.5-4.fc26.noarch
libobjc-7.3.1-2.fc26.s390x
dracut-046-8.git20180105.fc26.s390x
libseccomp-2.3.3-1.fc26.s390x
python-sphinx-locale-1.5.6-1.fc26.noarch
libxml2-devel-2.9.7-1.fc26.s390x
libseccomp-devel-2.3.3-1.fc26.s390x
fedora-upgrade-28.1-1.fc26.noarch
gcc-gfortran-7.3.1-2.fc26.s390x
gdb-8.0.1-36.fc26.s390x
unbound-libs-1.6.8-1.fc26.s390x
man-db-2.7.6.1-9.fc26.s390x
python2-rpm-macros-3-21.fc26.noarch
kernel-devel-4.15.4-200.fc26.s390x
sqlite-libs-3.20.1-2.fc26.s390x
python2-2.7.14-7.fc26.s390x
libkadm5-1.15.2-7.fc26.s390x
libcurl-7.53.1-16.fc26.s390x
net-snmp-agent-libs-5.7.3-27.fc26.s390x
p11-kit-trust-0.23.10-1.fc26.s390x
python3-koji-1.15.0-4.fc26.noarch
glusterfs-server-3.10.11-1.fc26.s390x
kernel-devel-4.15.12-201.fc26.s390x
pcre-utf16-8.41-6.fc26.s390x
jansson-2.11-1.fc26.s390x
python2-rpkg-1.52-1.fc26.noarch
pcre-static-8.41-6.fc26.s390x
systemtap-sdt-devel-3.2-7.fc26.s390x
libXfont-1.5.4-1.fc26.s390x
system-python-3.6.4-3.fc26.s390x
shared-mime-info-1.8-3.fc26.s390x
libpaper-1.1.24-21.fc26.s390x
python3-pbr-1.10.0-4.fc26.noarch
libcroco-0.6.12-1.fc26.s390x
libssh2-1.8.0-5.fc26.s390x
json-glib-1.2.6-1.fc26.s390x
libevent-2.0.22-3.fc26.s390x
gdk-pixbuf2-modules-2.36.9-1.fc26.s390x
colord-libs-1.3.5-1.fc26.s390x
python3-magic-5.30-11.fc26.noarch
python3-gobject-base-3.24.1-1.fc26.s390x
python3-pyroute2-0.4.13-1.fc26.noarch
python3-pysocks-1.6.7-1.fc26.noarch
python2-click-6.7-3.fc26.noarch
python-munch-2.1.0-2.fc26.noarch
python2-ply-3.9-3.fc26.noarch
python2-snowballstemmer-1.2.1-3.fc26.noarch
python-magic-5.30-11.fc26.noarch
python-beautifulsoup4-4.6.0-1.fc26.noarch
python2-gitdb-2.0.3-1.fc26.noarch
librados-devel-10.2.7-2.fc26.s390x
libcacard-2.5.3-1.fc26.s390x
libmodman-2.0.1-13.fc26.s390x
zziplib-0.13.62-8.fc26.s390x
lksctp-tools-1.0.16-6.fc26.s390x
procmail-3.22-44.fc26.s390x
libthai-0.1.25-2.fc26.s390x
libpipeline-1.4.1-3.fc26.s390x
python2-pycurl-7.43.0-8.fc26.s390x
deltarpm-3.6-19.fc26.s390x
subversion-libs-1.9.7-1.fc26.s390x
python-krbV-1.0.90-13.fc26.s390x
m17n-db-1.7.0-8.fc26.noarch
linux-atm-libs-2.5.1-17.fc26.s390x
python2-rpm-4.13.0.2-1.fc26.s390x
python2-librepo-1.8.0-1.fc26.s390x
qrencode-libs-3.4.4-1.fc26.s390x
s390utils-iucvterm-1.36.1-3.fc26.s390x
libsmartcols-2.30.2-1.fc26.s390x
dbus-1.11.18-1.fc26.s390x
systemd-udev-233-7.fc26.s390x
device-mapper-event-1.02.137-6.fc26.s390x
polkit-0.113-8.fc26.s390x
libwmf-lite-0.2.8.4-53.fc26.s390x
libXcomposite-0.4.4-9.fc26.s390x
at-spi2-atk-2.24.1-1.fc26.s390x
pango-1.40.12-1.fc26.s390x
texlive-metafont-bin-svn40987-33.20160520.fc26.2.s390x
texlive-url-svn32528.3.4-33.fc26.2.noarch
texlive-fp-svn15878.0-33.fc26.2.noarch
texlive-latex-fonts-svn28888.0-33.fc26.2.noarch
texlive-mptopdf-bin-svn18674.0-33.20160520.fc26.2.noarch
texlive-fancybox-svn18304.1.4-33.fc26.2.noarch
texlive-lua-alt-getopt-svn29349.0.7.0-33.fc26.2.noarch
texlive-tex-bin-svn40987-33.20160520.fc26.2.s390x
texlive-texconfig-svn40768-33.fc26.2.noarch
texlive-wasy2-ps-svn35830.0-33.fc26.2.noarch
texlive-psfrag-svn15878.3.04-33.fc26.2.noarch
texlive-helvetic-svn31835.0-33.fc26.2.noarch
texlive-times-svn35058.0-33.fc26.2.noarch
texlive-cite-svn36428.5.5-33.fc26.2.noarch
texlive-fancyhdr-svn15878.3.1-33.fc26.2.noarch
texlive-luatex-bin-svn41091-33.20160520.fc26.2.s390x
texlive-lm-math-svn36915.1.959-33.fc26.2.noarch
texlive-ntgclass-svn15878.2.1a-33.fc26.2.noarch
texlive-sansmath-svn17997.1.1-33.fc26.2.noarch
texlive-textcase-svn15878.0-33.fc26.2.noarch
texlive-unicode-data-svn39808-33.fc26.2.noarch
texlive-breakurl-svn29901.1.40-33.fc26.2.noarch
texlive-latex-svn40218-33.fc26.2.noarch
texlive-lualatex-math-svn40621-33.fc26.2.noarch
texlive-pst-coil-svn37377.1.07-33.fc26.2.noarch
texlive-pst-plot-svn41242-33.fc26.2.noarch
texlive-unicode-math-svn38462-33.fc26.2.noarch
texlive-pst-blur-svn15878.2.0-33.fc26.2.noarch
texlive-cm-super-svn15878.0-33.fc26.2.noarch
texlive-wasysym-svn15878.2.0-33.fc26.2.noarch
texlive-collection-fontsrecommended-svn35830.0-33.20160520.fc26.2.noarch
libXv-1.0.11-2.fc26.s390x
ghc-srpm-macros-1.4.2-5.fc26.noarch
latex2html-2017.2-2.fc26.noarch
libXau-devel-1.0.8-7.fc26.s390x
graphite2-devel-1.3.10-1.fc26.s390x
pixman-devel-0.34.0-3.fc26.s390x
wayland-protocols-devel-1.9-1.fc26.noarch
mesa-libGLES-devel-17.2.4-2.fc26.s390x
vte291-devel-0.48.4-1.fc26.s390x
ceph-devel-compat-10.2.7-2.fc26.s390x
lzo-devel-2.08-9.fc26.s390x
libiscsi-devel-1.15.0-3.fc26.s390x
avahi-autoipd-0.6.32-7.fc26.s390x
rpm-plugin-systemd-inhibit-4.13.0.2-1.fc26.s390x
python2-ndg_httpsclient-0.4.0-7.fc26.noarch
gettext-0.19.8.1-9.fc26.s390x
btrfs-progs-4.9.1-2.fc26.s390x
fedora-logos-26.0.1-1.fc26.s390x
dejagnu-1.6-2.fc26.noarch
libaio-devel-0.3.110-7.fc26.s390x
dos2unix-7.3.4-2.fc26.s390x
popt-1.16-12.fc26.s390x
tar-1.29-5.fc26.s390x
avahi-libs-0.6.32-7.fc26.s390x
m4-1.4.18-3.fc26.s390x
perl-Time-Local-1.250-2.fc26.noarch
libmetalink-0.1.3-2.fc26.s390x
jbigkit-libs-2.1-6.fc26.s390x
netpbm-10.80.00-2.fc26.s390x
perl-Digest-MD5-2.55-3.fc26.s390x
perl-Getopt-Long-2.49.1-2.fc26.noarch
libglvnd-opengl-1.0.0-1.fc26.s390x
libattr-devel-2.4.47-18.fc26.s390x
teckit-2.5.1-16.fc26.s390x
python3-six-1.10.0-9.fc26.noarch
python3-libcomps-0.1.8-3.fc26.s390x
python3-pyparsing-2.1.10-3.fc26.noarch
python2-markupsafe-0.23-13.fc26.s390x
python2-mock-2.0.0-4.fc26.noarch
python2-yubico-1.3.2-7.fc26.noarch
python2-smmap-2.0.3-1.fc26.noarch
librbd-devel-10.2.7-2.fc26.s390x
libnghttp2-1.21.1-1.fc26.s390x
ykpers-1.18.0-2.fc26.s390x
python3-librepo-1.8.0-1.fc26.s390x
geoclue2-2.4.5-4.fc26.s390x
initscripts-9.72-1.fc26.s390x
plymouth-0.9.3-0.7.20160620git0e65b86c.fc26.s390x
ebtables-2.0.10-22.fc26.s390x
gssproxy-0.7.0-9.fc26.s390x
libXext-1.3.3-5.fc26.s390x
mesa-libEGL-17.2.4-2.fc26.s390x
texlive-texlive.infra-bin-svn40312-33.20160520.fc26.2.s390x
texlive-thumbpdf-svn34621.3.16-33.fc26.2.noarch
texlive-carlisle-svn18258.0-33.fc26.2.noarch
texlive-gsftopk-svn40768-33.fc26.2.noarch
texlive-pdftex-svn41149-33.fc26.2.noarch
texlive-crop-svn15878.1.5-33.fc26.2.noarch
texlive-pxfonts-svn15878.0-33.fc26.2.noarch
texlive-enctex-svn34957.0-33.fc26.2.noarch
texlive-kastrup-svn15878.0-33.fc26.2.noarch
texlive-pspicture-svn15878.0-33.fc26.2.noarch
texlive-varwidth-svn24104.0.92-33.fc26.2.noarch
texlive-currfile-svn40725-33.fc26.2.noarch
texlive-pst-grad-svn15878.1.06-33.fc26.2.noarch
texlive-latex-bin-svn41438-33.fc26.2.noarch
texlive-ltxmisc-svn21927.0-33.fc26.2.noarch
lasi-1.1.2-7.fc26.s390x
adwaita-icon-theme-3.24.0-2.fc26.noarch
xz-devel-5.2.3-2.fc26.s390x
xorg-x11-fonts-Type1-7.5-17.fc26.noarch
libXi-devel-1.7.9-2.fc26.s390x
at-spi2-atk-devel-2.24.1-1.fc26.s390x
pango-devel-1.40.12-1.fc26.s390x
libcacard-devel-2.5.3-1.fc26.s390x
subversion-1.9.7-1.fc26.s390x
sudo-1.8.21p2-1.fc26.s390x
pykickstart-2.35-2.fc26.noarch
e2fsprogs-1.43.4-2.fc26.s390x
libbsd-0.8.3-3.fc26.s390x
c-ares-1.13.0-1.fc26.s390x
python2-pyxdg-0.25-12.fc26.noarch
valgrind-3.13.0-12.fc26.s390x
libwayland-server-1.13.0-3.fc26.s390x
dhcp-client-4.3.5-10.fc26.s390x
man-pages-4.09-4.fc26.noarch
libffi-devel-3.1-12.fc26.s390x
libxml2-2.9.7-1.fc26.s390x
kmod-25-1.fc26.s390x
dnf-plugins-core-2.1.5-4.fc26.noarch
kmod-libs-25-1.fc26.s390x
pigz-2.4-1.fc26.s390x
pkgconf-pkg-config-1.3.12-2.fc26.s390x
gcc-go-7.3.1-2.fc26.s390x
python-rpm-macros-3-21.fc26.noarch
perl-libs-5.24.3-396.fc26.s390x
glusterfs-api-3.10.11-1.fc26.s390x
git-core-doc-2.13.6-3.fc26.s390x
pcre-cpp-8.41-6.fc26.s390x
usermode-1.112-1.fc26.s390x
distribution-gpg-keys-1.19-1.fc26.noarch
perl-Test-Harness-3.41-1.fc26.noarch
krb5-devel-1.15.2-7.fc26.s390x
libvorbis-1.3.6-1.fc26.s390x
python2-configargparse-0.12.0-1.fc26.noarch
gpg-pubkey-a29cb19c-53bcbba6
quota-nls-4.03-9.fc26.noarch
xz-libs-5.2.3-2.fc26.s390x
gmp-6.1.2-4.fc26.s390x
file-5.30-11.fc26.s390x
libusbx-1.0.21-2.fc26.s390x
binutils-2.27-28.fc26.s390x
perl-HTTP-Tiny-0.070-2.fc26.noarch
xml-common-0.6.3-45.fc26.noarch
opus-1.2.1-1.fc26.s390x
flac-libs-1.3.2-2.fc26.s390x
libacl-devel-2.2.52-15.fc26.s390x
coreutils-common-8.27-7.fc26.s390x
cracklib-2.9.6-5.fc26.s390x
pyliblzma-0.5.3-17.fc26.s390x
libnotify-0.7.7-2.fc26.s390x
python3-idna-2.5-1.fc26.noarch
python3-pyOpenSSL-16.2.0-6.fc26.noarch
python2-pbr-1.10.0-4.fc26.noarch
pyusb-1.0.0-4.fc26.noarch
librbd1-10.2.7-2.fc26.s390x
libnfs-1.9.8-3.fc26.s390x
libsolv-0.6.30-2.fc26.s390x
python3-pycurl-7.43.0-8.fc26.s390x
libyubikey-1.13-3.fc26.s390x
rpmlint-1.10-5.fc26.noarch
python2-pygpgme-0.3-22.fc26.s390x
s390utils-base-1.36.1-3.fc26.s390x
ppp-2.4.7-11.fc26.s390x
s390utils-cpuplugd-1.36.1-3.fc26.s390x
libXrender-0.9.10-2.fc26.s390x
libglvnd-gles-1.0.0-1.fc26.s390x
texlive-texlive.infra-svn41280-33.fc26.2.noarch
texlive-lm-svn28119.2.004-33.fc26.2.noarch
texlive-babelbib-svn25245.1.31-33.fc26.2.noarch
texlive-index-svn24099.4.1beta-33.fc26.2.noarch
texlive-pdftex-bin-svn40987-33.20160520.fc26.2.s390x
texlive-csquotes-svn39538-33.fc26.2.noarch
texlive-rsfs-svn15878.0-33.fc26.2.noarch
texlive-etex-svn37057.0-33.fc26.2.noarch
texlive-knuth-lib-svn35820.0-33.fc26.2.noarch
texlive-pst-math-svn34786.0.63-33.fc26.2.noarch
texlive-utopia-svn15878.0-33.fc26.2.noarch
texlive-eso-pic-svn37925.2.0g-33.fc26.2.noarch
texlive-pst-fill-svn15878.1.01-33.fc26.2.noarch
texlive-latex-bin-bin-svn14050.0-33.20160520.fc26.2.noarch
texlive-jknapltx-svn19440.0-33.fc26.2.noarch
texlive-collection-latexrecommended-svn35765.0-33.20160520.fc26.2.noarch
adwaita-cursor-theme-3.24.0-2.fc26.noarch
xorg-x11-fonts-ISO8859-1-100dpi-7.5-17.fc26.noarch
libXcomposite-devel-0.4.4-9.fc26.s390x
at-spi2-core-devel-2.24.1-1.fc26.s390x
harfbuzz-devel-1.4.4-1.fc26.s390x
rpmdevtools-8.10-2.fc26.noarch
texi2html-5.0-5.fc26.noarch
libnfs-devel-1.9.8-3.fc26.s390x
firewalld-0.4.4.5-1.fc26.noarch
wpa_supplicant-2.6-12.fc26.s390x
newt-python-0.52.20-1.fc26.s390x
perl-Mozilla-CA-20160104-4.fc26.noarch
pth-2.0.7-28.fc26.s390x
python3-pyxdg-0.25-12.fc26.noarch
timedatex-0.4-3.fc26.s390x
libjpeg-turbo-1.5.3-1.fc26.s390x
dnf-yum-2.7.5-2.fc26.noarch
libuv-devel-1.11.0-1.fc26.s390x
libstdc++-7.3.1-2.fc26.s390x
libgo-7.3.1-2.fc26.s390x
python3-dnf-plugins-core-2.1.5-4.fc26.noarch
gtk3-3.22.21-3.fc26.s390x
perl-threads-2.21-1.fc26.s390x
pkgconf-m4-1.3.12-2.fc26.noarch
gtk3-devel-3.22.21-3.fc26.s390x
gcc-objc-7.3.1-2.fc26.s390x
nss-util-3.36.0-1.0.fc26.s390x
python2-koji-1.15.0-4.fc26.noarch
kernel-modules-4.15.12-201.fc26.s390x
elfutils-libelf-devel-0.170-4.fc26.s390x
selinux-policy-3.13.1-260.20.fc26.noarch
mock-core-configs-28.3-1.fc26.noarch
glusterfs-api-devel-3.10.11-1.fc26.s390x
krb5-workstation-1.15.2-7.fc26.s390x
libsss_sudo-1.16.1-1.fc26.s390x
mailcap-2.1.48-1.fc26.noarch
python-async-0.6.1-9.fc22.s390x
poppler-data-0.4.7-7.fc26.noarch
ocaml-srpm-macros-4-2.fc26.noarch
libuuid-2.30.2-1.fc26.s390x
libgpg-error-1.25-2.fc26.s390x
graphite2-1.3.10-1.fc26.s390x
perl-Text-Tabs+Wrap-2013.0523-366.fc26.noarch
perl-Error-0.17024-8.fc26.noarch
which-2.21-2.fc26.s390x
libXau-1.0.8-7.fc26.s390x
orc-0.4.27-1.fc26.s390x
perl-Pod-Perldoc-3.28-1.fc26.noarch
libsndfile-1.0.28-6.fc26.s390x
gzip-1.8-2.fc26.s390x
python-ipaddress-1.0.16-4.fc26.noarch
yum-metadata-parser-1.1.4-18.fc26.s390x
python3-dbus-1.2.4-6.fc26.s390x
python3-cryptography-2.0.2-2.fc26.s390x
python3-kickstart-2.35-2.fc26.noarch
python2-imagesize-0.7.1-5.fc26.noarch
python2-jinja2-2.9.6-1.fc26.noarch
libradosstriper-devel-10.2.7-2.fc26.s390x
soundtouch-1.9.2-4.fc26.s390x
libndp-1.6-2.fc26.s390x
rpm-4.13.0.2-1.fc26.s390x
rest-0.8.0-2.fc26.s390x
libvisual-0.4.0-21.fc26.s390x
python2-hawkey-0.11.1-1.fc26.s390x
fakeroot-libs-1.22-1.fc26.s390x
device-mapper-event-libs-1.02.137-6.fc26.s390x
cyrus-sasl-2.1.26-32.fc26.s390x
cronie-anacron-1.5.1-5.fc26.s390x
libpath_utils-0.2.1-34.fc26.s390x
libX11-common-1.6.5-2.fc26.noarch
libXft-2.3.2-5.fc26.s390x
gtk2-2.24.31-4.fc26.s390x
texlive-etoolbox-svn38031.2.2a-33.fc26.2.noarch
texlive-multido-svn18302.1.42-33.fc26.2.noarch
texlive-glyphlist-svn28576.0-33.fc26.2.noarch
texlive-setspace-svn24881.6.7a-33.fc26.2.noarch
texlive-mathtools-svn38833-33.fc26.2.noarch
texlive-ncntrsbk-svn31835.0-33.fc26.2.noarch
texlive-dvisvgm-def-svn41011-33.fc26.2.noarch
texlive-ifetex-svn24853.1.2-33.fc26.2.noarch
texlive-parskip-svn19963.2.0-33.fc26.2.noarch
texlive-bera-svn20031.0-33.fc26.2.noarch
texlive-pgf-svn40966-33.fc26.2.noarch
texlive-auto-pst-pdf-svn23723.0.6-33.fc26.2.noarch
texlive-ctable-svn38672-33.fc26.2.noarch
texlive-typehtml-svn17134.0-33.fc26.2.noarch
mesa-libGLES-17.2.4-2.fc26.s390x
vte291-0.48.4-1.fc26.s390x
libcephfs_jni1-10.2.7-2.fc26.s390x
bzip2-devel-1.0.6-22.fc26.s390x
expat-devel-2.2.4-1.fc26.s390x
libsepol-devel-2.6-2.fc26.s390x
glib2-static-2.52.3-2.fc26.s390x
virglrenderer-devel-0.6.0-1.20170210git76b3da97b.fc26.s390x
parted-3.2-24.fc26.s390x
python3-beautifulsoup4-4.6.0-1.fc26.noarch
python-bunch-1.0.1-10.fc26.noarch
lz4-1.8.0-1.fc26.s390x
openssh-clients-7.5p1-4.fc26.s390x
chrony-3.2-1.fc26.s390x
dnf-conf-2.7.5-2.fc26.noarch
bodhi-client-2.12.2-3.fc26.noarch
libuv-1.11.0-1.fc26.s390x
glibc-2.25-13.fc26.s390x
libgomp-7.3.1-2.fc26.s390x
cmake-rpm-macros-3.10.1-11.fc26.noarch
gtk-update-icon-cache-3.22.21-3.fc26.s390x
pcre2-utf32-10.23-13.fc26.s390x
kernel-modules-4.15.4-200.fc26.s390x
webkitgtk4-2.18.6-1.fc26.s390x
libstdc++-static-7.3.1-2.fc26.s390x
rsync-3.1.3-2.fc26.s390x
nspr-4.19.0-1.fc26.s390x
nss-util-devel-3.36.0-1.0.fc26.s390x
kernel-core-4.15.12-201.fc26.s390x
glusterfs-extra-xlators-3.10.11-1.fc26.s390x
vim-filesystem-8.0.1553-1.fc26.noarch
systemtap-client-3.2-7.fc26.s390x
net-snmp-5.7.3-27.fc26.s390x
mailx-12.5-25.fc26.s390x
mpfr-3.1.6-1.fc26.s390x
libzip-devel-1.3.0-1.fc26.s390x
hawkey-0.6.4-3.fc25.s390x
perl-srpm-macros-1-21.fc26.noarch
expat-2.2.4-1.fc26.s390x
chkconfig-1.10-1.fc26.s390x
findutils-4.6.0-12.fc26.s390x
mesa-libwayland-egl-17.2.4-2.fc26.s390x
procps-ng-3.3.10-13.fc26.s390x
mesa-libglapi-17.2.4-2.fc26.s390x
perl-Unicode-Normalize-1.25-366.fc26.s390x
perl-IO-Socket-IP-0.39-1.fc26.noarch
hunspell-en-US-0.20140811.1-6.fc26.noarch
libxcb-1.12-3.fc26.s390x
perl-Pod-Escapes-1.07-366.fc26.noarch
perl-Pod-Usage-1.69-2.fc26.noarch
libtheora-1.1.1-15.fc26.s390x
tcp_wrappers-7.6-85.fc26.s390x
coreutils-8.27-7.fc26.s390x
libmount-2.30.2-1.fc26.s390x
python2-iniparse-0.4-24.fc26.noarch
python2-decorator-4.0.11-2.fc26.noarch
ModemManager-glib-1.6.10-1.fc26.s390x
python3-decorator-4.0.11-2.fc26.noarch
python3-cffi-1.9.1-2.fc26.s390x
python-bugzilla-cli-2.1.0-1.fc26.noarch
python2-funcsigs-1.0.2-5.fc26.noarch
python2-babel-2.3.4-5.fc26.noarch
python-bugzilla-2.1.0-1.fc26.noarch
libradosstriper1-10.2.7-2.fc26.s390x
snappy-1.1.4-3.fc26.s390x
libmpcdec-1.2.6-17.fc26.s390x
rpm-libs-4.13.0.2-1.fc26.s390x
python-urlgrabber-3.10.1-11.fc26.noarch
sysfsutils-2.1.0-20.fc26.s390x
python3-hawkey-0.11.1-1.fc26.s390x
iputils-20161105-5.fc26.s390x
plymouth-scripts-0.9.3-0.7.20160620git0e65b86c.fc26.s390x
cronie-1.5.1-5.fc26.s390x
libini_config-1.3.1-34.fc26.s390x
libX11-1.6.5-2.fc26.s390x
libglvnd-egl-1.0.0-1.fc26.s390x
texlive-kpathsea-svn41139-33.fc26.2.noarch
texlive-thumbpdf-bin-svn6898.0-33.20160520.fc26.2.noarch
texlive-subfig-svn15878.1.3-33.fc26.2.noarch
texlive-gsftopk-bin-svn40473-33.20160520.fc26.2.s390x
texlive-tex-ini-files-svn40533-33.fc26.2.noarch
texlive-qstest-svn15878.0-33.fc26.2.noarch
texlive-palatino-svn31835.0-33.fc26.2.noarch
texlive-ec-svn25033.1.0-33.fc26.2.noarch
texlive-iftex-svn29654.0.2-33.fc26.2.noarch
texlive-pslatex-svn16416.0-33.fc26.2.noarch
texlive-algorithms-svn38085.0.1-33.fc26.2.noarch
texlive-filehook-svn24280.0.5d-33.fc26.2.noarch
texlive-pst-node-svn40743-33.fc26.2.noarch
texlive-rotating-svn16832.2.16b-33.fc26.2.noarch
texlive-seminar-svn34011.1.62-33.fc26.2.noarch
libuuid-devel-2.30.2-1.fc26.s390x
libXinerama-devel-1.1.3-7.fc26.s390x
emacs-common-25.3-3.fc26.s390x
fedora-packager-0.6.0.1-2.fc26.noarch
snappy-devel-1.1.4-3.fc26.s390x
authconfig-7.0.1-2.fc26.s390x
newt-python3-0.52.20-1.fc26.s390x
python-decoratortools-1.8-13.fc26.noarch
python-systemd-doc-234-1.fc26.s390x
openssl-libs-1.1.0g-1.fc26.s390x
lsof-4.89-5.fc26.s390x
glibc-all-langpacks-2.25-13.fc26.s390x
audit-libs-2.8.2-1.fc26.s390x
gcc-7.3.1-2.fc26.s390x
pcre2-utf16-10.23-13.fc26.s390x
kernel-core-4.15.4-200.fc26.s390x
dracut-config-rescue-046-8.git20180105.fc26.s390x
webkitgtk4-plugin-process-gtk2-2.18.6-1.fc26.s390x
perl-Time-HiRes-1.9753-1.fc26.s390x
haveged-1.9.1-6.fc26.s390x
p11-kit-0.23.10-1.fc26.s390x
boost-system-1.63.0-11.fc26.s390x
glusterfs-fuse-3.10.11-1.fc26.s390x
vim-common-8.0.1553-1.fc26.s390x
systemtap-devel-3.2-7.fc26.s390x
perl-SelfLoader-1.23-396.fc26.noarch
nss-tools-3.36.0-1.0.fc26.s390x
libwebp-0.6.1-8.fc26.s390x
python3-configargparse-0.12.0-1.fc26.noarch
gpg-pubkey-a0a7badb-52844296
gpg-pubkey-e372e838-56fd7943
gpg-pubkey-3b921d09-57a87096
google-roboto-slab-fonts-1.100263-0.5.20150923git.fc26.noarch
libreport-filesystem-2.9.1-3.fc26.s390x
libcom_err-1.43.4-2.fc26.s390x
libffi-3.1-12.fc26.s390x
keyutils-libs-1.5.10-1.fc26.s390x
diffutils-3.5-3.fc26.s390x
apr-util-1.5.4-6.fc26.s390x
bluez-libs-5.46-6.fc26.s390x
libksba-1.3.5-3.fc26.s390x
ncurses-6.0-8.20170212.fc26.s390x
libteam-1.27-1.fc26.s390x
perl-Fedora-VSP-0.001-5.fc26.noarch
libusb-0.1.5-8.fc26.s390x
acl-2.2.52-15.fc26.s390x
dwz-0.12-3.fc26.s390x
libblkid-2.30.2-1.fc26.s390x
polkit-libs-0.113-8.fc26.s390x
dbus-python-1.2.4-6.fc26.s390x
gts-0.7.6-30.20121130.fc26.s390x
libfdisk-2.30.2-1.fc26.s390x
python3-pycparser-2.14-10.fc26.noarch
python3-bugzilla-2.1.0-1.fc26.noarch
python2-docutils-0.13.1-4.fc26.noarch
python2-requests-2.13.0-1.fc26.noarch
libcephfs-devel-10.2.7-2.fc26.s390x
ncurses-c++-libs-6.0-8.20170212.fc26.s390x
GeoIP-1.6.11-1.fc26.s390x
liblockfile-1.09-5.fc26.s390x
rpm-plugin-selinux-4.13.0.2-1.fc26.s390x
libsysfs-2.1.0-20.fc26.s390x
libdnf-0.11.1-1.fc26.s390x
mesa-libgbm-17.2.4-2.fc26.s390x
lvm2-libs-2.02.168-6.fc26.s390x
libXfixes-5.0.3-2.fc26.s390x
brlapi-0.6.6-5.fc26.s390x
texlive-metafont-svn40793-33.fc26.2.noarch
texlive-graphics-cfg-svn40269-33.fc26.2.noarch
texlive-mptopdf-svn41282-33.fc26.2.noarch
texlive-makeindex-bin-svn40473-33.20160520.fc26.2.s390x
texlive-texlive-scripts-bin-svn29741.0-33.20160520.fc26.2.noarch
texlive-sauerj-svn15878.0-33.fc26.2.noarch
texlive-txfonts-svn15878.0-33.fc26.2.noarch
texlive-filecontents-svn24250.1.3-33.fc26.2.noarch
texlive-lualibs-svn40370-33.fc26.2.noarch
texlive-section-svn20180.0-33.fc26.2.noarch
texlive-ucharcat-svn38907-33.fc26.2.noarch
texlive-hyperref-svn41396-33.fc26.2.noarch
texlive-pst-3d-svn17257.1.10-33.fc26.2.noarch
texlive-oberdiek-svn41346-33.fc26.2.noarch
texlive-ae-svn15878.1.4-33.fc26.2.noarch
texlive-collection-basic-svn41149-33.20160520.fc26.2.noarch
gnat-srpm-macros-4-2.fc26.noarch
glib2-devel-2.52.3-2.fc26.s390x
netpbm-progs-10.80.00-2.fc26.s390x
libXxf86vm-devel-1.1.4-4.fc26.s390x
nettle-devel-3.3-2.fc26.s390x
cairo-gobject-devel-1.14.10-1.fc26.s390x
fedora-rpm-macros-26-2.fc26.noarch
libidn-devel-1.33-2.fc26.s390x
s390utils-1.36.1-3.fc26.s390x
libtool-2.4.6-17.fc26.s390x
python3-cssselect-0.9.2-4.fc26.noarch
python2-cssselect-0.9.2-4.fc26.noarch
bison-3.0.4-6.fc26.s390x
rootfiles-8.1-20.fc26.noarch
python3-urllib3-1.20-2.fc26.noarch
libgcc-7.3.1-2.fc26.s390x
python3-distro-1.2.0-1.fc26.noarch
libnfsidmap-2.2.1-4.rc2.fc26.s390x
kernel-4.15.4-200.fc26.s390x
glibc-static-2.25-13.fc26.s390x
xapian-core-libs-1.4.5-1.fc26.s390x
elfutils-libelf-0.170-4.fc26.s390x
nss-3.36.0-1.0.fc26.s390x
nss-softokn-freebl-devel-3.36.0-1.0.fc26.s390x
koji-1.15.0-4.fc26.noarch
perl-Git-2.13.6-3.fc26.noarch
elfutils-default-yama-scope-0.170-4.fc26.noarch
selinux-policy-targeted-3.13.1-260.20.fc26.noarch
curl-7.53.1-16.fc26.s390x
publicsuffix-list-dafsa-20180223-1.fc26.noarch
python3-funcsigs-1.0.2-5.fc26.noarch
=== TEST BEGIN ===
Using CC: /home/fam/bin/cc
Install prefix    /var/tmp/patchew-tester-tmp-yhgce4ah/src/install
BIOS directory    /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/share/qemu
firmware path     /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/share/qemu-firmware
binary directory  /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/bin
library directory /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/lib
module directory  /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/lib/qemu
libexec directory /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/libexec
include directory /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/include
config directory  /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/etc
local state directory   /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/var
Manual directory  /var/tmp/patchew-tester-tmp-yhgce4ah/src/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /var/tmp/patchew-tester-tmp-yhgce4ah/src
GIT binary        git
GIT submodules    ui/keycodemapdb capstone
C compiler        /home/fam/bin/cc
Host C compiler   cc
C++ compiler      c++
Objective-C compiler /home/fam/bin/cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1   -Werror -DHAS_LIBSSH2_SFTP_FSYNC -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include  -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1     -I/usr/include/libpng16  -I/usr/include/libdrm  -I$(SRC_PATH)/capstone/include
LDFLAGS           -Wl,--warn-common -m64 -g 
QEMU_LDFLAGS       
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          s390x
host big endian   yes
target list       aarch64-softmmu alpha-softmmu arm-softmmu cris-softmmu hppa-softmmu i386-softmmu lm32-softmmu m68k-softmmu microblazeel-softmmu microblaze-softmmu mips64el-softmmu mips64-softmmu mipsel-softmmu mips-softmmu moxie-softmmu nios2-softmmu or1k-softmmu ppc64-softmmu ppcemb-softmmu ppc-softmmu riscv32-softmmu riscv64-softmmu s390x-softmmu sh4eb-softmmu sh4-softmmu sparc64-softmmu sparc-softmmu tricore-softmmu unicore32-softmmu x86_64-softmmu xtensaeb-softmmu xtensa-softmmu aarch64_be-linux-user aarch64-linux-user alpha-linux-user armeb-linux-user arm-linux-user cris-linux-user hppa-linux-user i386-linux-user m68k-linux-user microblazeel-linux-user microblaze-linux-user mips64el-linux-user mips64-linux-user mipsel-linux-user mips-linux-user mipsn32el-linux-user mipsn32-linux-user nios2-linux-user or1k-linux-user ppc64abi32-linux-user ppc64le-linux-user ppc64-linux-user ppc-linux-user riscv32-linux-user riscv64-linux-user s390x-linux-user sh4eb-linux-user sh4-linux-user sparc32plus-linux-user sparc64-linux-user sparc-linux-user tilegx-linux-user x86_64-linux-user xtensaeb-linux-user xtensa-linux-user
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
SDL support       yes (2.0.7)
GTK support       yes (3.22.21)
GTK GL support    yes
VTE support       yes (0.48.4)
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    yes
virgl support     yes (0.6.0)
curl support      yes
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    yes
Multipath support no
VNC support       yes
VNC SASL support  yes
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    yes
bluez  support    yes
Documentation     yes
PIE               no
vde support       no
netmap support    no
Linux AIO support yes
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
HAX support       no
HVF support       no
WHPX support      no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
malloc trim support yes
RDMA support      no
fdt support       system
membarrier        no
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
posix_memalign    yes
libcap-ng support yes
vhost-net support yes
vhost-crypto support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends    log
spice support     no 
rbd support       yes
xfsctl support    no
smartcard support yes
libusb            yes
usb net redir     yes
OpenGL support    yes
OpenGL dmabufs    yes
libiscsi support  yes
libnfs support    yes
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   yes
coroutine backend ucontext
coroutine pool    yes
debug stack usage no
mutex debugging   no
crypto afalg      no
GlusterFS support yes
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
TPM emulator      yes
QOM debugging     yes
Live block migration yes
lzo support       yes
snappy support    yes
bzip2 support     yes
NUMA host support no
libxml2           yes
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
VxHS block device no
capstone          git
docker            no

NOTE: cross-compilers enabled:  '/home/fam/bin/cc' '/home/fam/bin/cc'
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak.tmp
  GEN     arm-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak
  GEN     hppa-softmmu/config-devices.mak.tmp
  GEN     arm-softmmu/config-devices.mak
  GEN     alpha-softmmu/config-devices.mak
  GEN     i386-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak
  GEN     lm32-softmmu/config-devices.mak.tmp
  GEN     m68k-softmmu/config-devices.mak.tmp
  GEN     hppa-softmmu/config-devices.mak
  GEN     lm32-softmmu/config-devices.mak
  GEN     microblazeel-softmmu/config-devices.mak.tmp
  GEN     m68k-softmmu/config-devices.mak
  GEN     microblaze-softmmu/config-devices.mak.tmp
  GEN     mips64el-softmmu/config-devices.mak.tmp
  GEN     i386-softmmu/config-devices.mak
  GEN     mips64-softmmu/config-devices.mak.tmp
  GEN     microblaze-softmmu/config-devices.mak
  GEN     mipsel-softmmu/config-devices.mak.tmp
  GEN     microblazeel-softmmu/config-devices.mak
  GEN     mips-softmmu/config-devices.mak.tmp
  GEN     mips64el-softmmu/config-devices.mak
  GEN     moxie-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak
  GEN     mipsel-softmmu/config-devices.mak
  GEN     nios2-softmmu/config-devices.mak.tmp
  GEN     moxie-softmmu/config-devices.mak
  GEN     mips-softmmu/config-devices.mak
  GEN     or1k-softmmu/config-devices.mak.tmp
  GEN     ppc64-softmmu/config-devices.mak.tmp
  GEN     ppcemb-softmmu/config-devices.mak.tmp
  GEN     nios2-softmmu/config-devices.mak
  GEN     or1k-softmmu/config-devices.mak
  GEN     ppc-softmmu/config-devices.mak.tmp
  GEN     riscv32-softmmu/config-devices.mak.tmp
  GEN     ppcemb-softmmu/config-devices.mak
  GEN     riscv32-softmmu/config-devices.mak
  GEN     riscv64-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak
  GEN     ppc64-softmmu/config-devices.mak
  GEN     s390x-softmmu/config-devices.mak.tmp
  GEN     sh4eb-softmmu/config-devices.mak.tmp
  GEN     sh4-softmmu/config-devices.mak.tmp
  GEN     riscv64-softmmu/config-devices.mak
  GEN     sparc64-softmmu/config-devices.mak.tmp
  GEN     s390x-softmmu/config-devices.mak
  GEN     sparc-softmmu/config-devices.mak.tmp
  GEN     sh4eb-softmmu/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak.tmp
  GEN     sparc-softmmu/config-devices.mak
  GEN     unicore32-softmmu/config-devices.mak.tmp
  GEN     sparc64-softmmu/config-devices.mak
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     xtensaeb-softmmu/config-devices.mak.tmp
  GEN     tricore-softmmu/config-devices.mak
  GEN     unicore32-softmmu/config-devices.mak
  GEN     xtensa-softmmu/config-devices.mak.tmp
  GEN     aarch64_be-linux-user/config-devices.mak.tmp
  GEN     xtensaeb-softmmu/config-devices.mak
  GEN     aarch64-linux-user/config-devices.mak.tmp
  GEN     aarch64_be-linux-user/config-devices.mak
  GEN     xtensa-softmmu/config-devices.mak
  GEN     aarch64-linux-user/config-devices.mak
  GEN     alpha-linux-user/config-devices.mak.tmp
  GEN     armeb-linux-user/config-devices.mak.tmp
  GEN     arm-linux-user/config-devices.mak.tmp
  GEN     x86_64-softmmu/config-devices.mak
  GEN     armeb-linux-user/config-devices.mak
  GEN     cris-linux-user/config-devices.mak.tmp
  GEN     alpha-linux-user/config-devices.mak
  GEN     arm-linux-user/config-devices.mak
  GEN     hppa-linux-user/config-devices.mak.tmp
  GEN     i386-linux-user/config-devices.mak.tmp
  GEN     m68k-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak
  GEN     i386-linux-user/config-devices.mak
  GEN     cris-linux-user/config-devices.mak
  GEN     microblazeel-linux-user/config-devices.mak.tmp
  GEN     microblaze-linux-user/config-devices.mak.tmp
  GEN     mips64el-linux-user/config-devices.mak.tmp
  GEN     m68k-linux-user/config-devices.mak
  GEN     mips64-linux-user/config-devices.mak.tmp
  GEN     microblazeel-linux-user/config-devices.mak
  GEN     microblaze-linux-user/config-devices.mak
  GEN     mipsel-linux-user/config-devices.mak.tmp
  GEN     mips64el-linux-user/config-devices.mak
  GEN     mips-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak
  GEN     mipsn32el-linux-user/config-devices.mak.tmp
  GEN     mipsn32-linux-user/config-devices.mak.tmp
  GEN     mipsel-linux-user/config-devices.mak
  GEN     mips-linux-user/config-devices.mak
  GEN     mipsn32el-linux-user/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak.tmp
  GEN     mipsn32-linux-user/config-devices.mak
  GEN     ppc64abi32-linux-user/config-devices.mak.tmp
  GEN     ppc64le-linux-user/config-devices.mak.tmp
  GEN     or1k-linux-user/config-devices.mak
  GEN     nios2-linux-user/config-devices.mak
  GEN     ppc64abi32-linux-user/config-devices.mak
  GEN     ppc64-linux-user/config-devices.mak.tmp
  GEN     ppc-linux-user/config-devices.mak.tmp
  GEN     riscv32-linux-user/config-devices.mak.tmp
  GEN     ppc64le-linux-user/config-devices.mak
  GEN     riscv32-linux-user/config-devices.mak
  GEN     ppc64-linux-user/config-devices.mak
  GEN     riscv64-linux-user/config-devices.mak.tmp
  GEN     ppc-linux-user/config-devices.mak
  GEN     sh4eb-linux-user/config-devices.mak.tmp
  GEN     sh4-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak.tmp
  GEN     riscv64-linux-user/config-devices.mak
  GEN     s390x-linux-user/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak.tmp
  GEN     sh4eb-linux-user/config-devices.mak
  GEN     sparc64-linux-user/config-devices.mak.tmp
  GEN     sh4-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak.tmp
  GEN     tilegx-linux-user/config-devices.mak.tmp
  GEN     sparc32plus-linux-user/config-devices.mak
  GEN     sparc64-linux-user/config-devices.mak
  GEN     x86_64-linux-user/config-devices.mak.tmp
  GEN     tilegx-linux-user/config-devices.mak
  GEN     xtensaeb-linux-user/config-devices.mak.tmp
  GEN     sparc-linux-user/config-devices.mak
  GEN     xtensa-linux-user/config-devices.mak.tmp
  GEN     x86_64-linux-user/config-devices.mak
  GEN     xtensaeb-linux-user/config-devices.mak
  GEN     xtensa-linux-user/config-devices.mak
  GEN     config-host.h
  GIT     ui/keycodemapdb capstone
  GEN     qemu-options.def
  GEN     qapi-gen
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.h
  GEN     trace/generated-helpers.c
  GEN     module_block.h
Submodule 'capstone' (git://git.qemu.org/capstone.git) registered for path 'capstone'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
  GEN     tests/test-qapi-gen
Cloning into '/var/tmp/patchew-tester-tmp-yhgce4ah/src/capstone'...
  GEN     trace-root.h
  GEN     accel/kvm/trace.h
  GEN     accel/tcg/trace.h
  GEN     audio/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     crypto/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/display/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/hppa/trace.h
  GEN     hw/i2c/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/ide/trace.h
  GEN     hw/input/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/misc/macio/trace.h
  GEN     hw/net/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/pci-host/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/rdma/trace.h
  GEN     hw/rdma/vmw/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/sparc64/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/tpm/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/xen/trace.h
  GEN     io/trace.h
  GEN     linux-user/trace.h
  GEN     migration/trace.h
  GEN     nbd/trace.h
  GEN     net/trace.h
  GEN     qapi/trace.h
  GEN     qom/trace.h
  GEN     scsi/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/ppc/trace.h
  GEN     target/s390x/trace.h
  GEN     target/sparc/trace.h
  GEN     ui/trace.h
  GEN     trace-root.c
  GEN     util/trace.h
  GEN     accel/kvm/trace.c
  GEN     accel/tcg/trace.c
  GEN     audio/trace.c
  GEN     block/trace.c
  GEN     chardev/trace.c
  GEN     crypto/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/alpha/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/display/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/hppa/trace.c
  GEN     hw/i2c/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/ide/trace.c
  GEN     hw/input/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/misc/macio/trace.c
  GEN     hw/net/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/pci-host/trace.c
  GEN     hw/rdma/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/rdma/vmw/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/sparc64/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/tpm/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/xen/trace.c
  GEN     io/trace.c
  GEN     linux-user/trace.c
  GEN     migration/trace.c
  GEN     nbd/trace.c
  GEN     net/trace.c
  GEN     qapi/trace.c
  GEN     qom/trace.c
  GEN     scsi/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/ppc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/sparc/trace.c
  GEN     ui/trace.c
  GEN     util/trace.c
  GEN     config-all-devices.mak
Cloning into '/var/tmp/patchew-tester-tmp-yhgce4ah/src/ui/keycodemapdb'...
  GEN     ui/input-keymap-atset1-to-qcode.c
  GEN     ui/input-keymap-linux-to-qcode.c
  GEN     ui/input-keymap-qcode-to-atset1.c
  GEN     ui/input-keymap-qcode-to-atset2.c
  GEN     ui/input-keymap-qcode-to-atset3.c
  GEN     ui/input-keymap-qcode-to-linux.c
  GEN     ui/input-keymap-qcode-to-qnum.c
  GEN     ui/input-keymap-qcode-to-sun.c
  GEN     ui/input-keymap-qnum-to-qcode.c
  GEN     ui/input-keymap-usb-to-qcode.c
  GEN     ui/input-keymap-win32-to-qcode.c
  CC      cs.o
  GEN     ui/input-keymap-x11-to-qcode.c
  GEN     ui/input-keymap-xorgevdev-to-qcode.c
  CC      utils.o
  CC      SStream.o
  CC      MCInstrDesc.o
  CC      MCRegisterInfo.o
  CC      arch/ARM/ARMDisassembler.o
  CC      arch/ARM/ARMInstPrinter.o
  GEN     ui/input-keymap-xorgkbd-to-qcode.c
  GEN     ui/input-keymap-xorgxquartz-to-qcode.c
  CC      arch/ARM/ARMMapping.o
  GEN     ui/input-keymap-xorgxwin-to-qcode.c
  GEN     ui/input-keymap-osx-to-qcode.c
  CC      arch/ARM/ARMModule.o
  CC      arch/AArch64/AArch64BaseInfo.o
  CC      arch/AArch64/AArch64Disassembler.o
  CC      arch/AArch64/AArch64InstPrinter.o
  CC      arch/AArch64/AArch64Mapping.o
  CC      arch/AArch64/AArch64Module.o
  CC      arch/Mips/MipsDisassembler.o
  CC      arch/Mips/MipsInstPrinter.o
  CC      arch/Mips/MipsMapping.o
  CC      arch/Mips/MipsModule.o
  CC      arch/PowerPC/PPCDisassembler.o
  CC      arch/PowerPC/PPCInstPrinter.o
  CC      arch/PowerPC/PPCMapping.o
  CC      arch/PowerPC/PPCModule.o
  CC      arch/Sparc/SparcDisassembler.o
  CC      arch/Sparc/SparcInstPrinter.o
  CC      arch/Sparc/SparcMapping.o
  CC      arch/Sparc/SparcModule.o
  CC      arch/SystemZ/SystemZDisassembler.o
  CC      arch/SystemZ/SystemZInstPrinter.o
  CC      arch/SystemZ/SystemZMapping.o
  CC      arch/SystemZ/SystemZModule.o
  CC      arch/SystemZ/SystemZMCTargetDesc.o
  CC      arch/X86/X86DisassemblerDecoder.o
  CC      arch/X86/X86Disassembler.o
  CC      arch/X86/X86IntelInstPrinter.o
  CC      arch/X86/X86ATTInstPrinter.o
  CC      arch/X86/X86Mapping.o
  CC      arch/X86/X86Module.o
  CC      arch/XCore/XCoreDisassembler.o
  CC      arch/XCore/XCoreInstPrinter.o
  CC      arch/XCore/XCoreMapping.o
  CC      arch/XCore/XCoreModule.o
  CC      MCInst.o
  AR      libcapstone.a
ar: creating /var/tmp/patchew-tester-tmp-yhgce4ah/src/build/capstone/libcapstone.a
  GEN     docs/version.texi
  GEN     qemu-options.texi
  CC      tests/qemu-iotests/socket_scm_helper.o
  GEN     qemu-monitor.texi
  GEN     qemu-img-cmds.texi
  GEN     qemu-monitor-info.texi
  GEN     qemu-img.1
  GEN     qemu-nbd.8
  GEN     qemu-ga.8
  GEN     qga/qapi-generated/qapi-gen
  GEN     docs/qemu-block-drivers.7
  GEN     fsdev/virtfs-proxy-helper.1
  CC      qapi/qapi-builtin-types.o
  CC      qapi/qapi-types.o
  CC      qapi/qapi-types-block-core.o
  CC      qapi/qapi-types-block.o
  CC      qapi/qapi-types-char.o
  CC      qapi/qapi-types-common.o
  CC      qapi/qapi-types-crypto.o
  CC      qapi/qapi-types-introspect.o
  CC      qapi/qapi-types-job.o
  CC      qapi/qapi-types-migration.o
  CC      qapi/qapi-types-misc.o
  CC      qapi/qapi-types-net.o
  CC      qapi/qapi-types-rocker.o
  CC      qapi/qapi-types-run-state.o
  CC      qapi/qapi-types-sockets.o
  CC      qapi/qapi-types-tpm.o
  CC      qapi/qapi-types-trace.o
  CC      qapi/qapi-types-transaction.o
  CC      qapi/qapi-types-ui.o
  CC      qapi/qapi-builtin-visit.o
  CC      qapi/qapi-visit.o
  CC      qapi/qapi-visit-block-core.o
  CC      qapi/qapi-visit-block.o
  CC      qapi/qapi-visit-char.o
  CC      qapi/qapi-visit-common.o
  CC      qapi/qapi-visit-crypto.o
  CC      qapi/qapi-visit-introspect.o
  CC      qapi/qapi-visit-job.o
  CC      qapi/qapi-visit-migration.o
  CC      qapi/qapi-visit-misc.o
  CC      qapi/qapi-visit-net.o
  CC      qapi/qapi-visit-rocker.o
  CC      qapi/qapi-visit-run-state.o
  CC      qapi/qapi-visit-sockets.o
  CC      qapi/qapi-visit-tpm.o
  CC      qapi/qapi-visit-trace.o
  CC      qapi/qapi-visit-transaction.o
  CC      qapi/qapi-visit-ui.o
  CC      qapi/qapi-events.o
  CC      qapi/qapi-events-block-core.o
  CC      qapi/qapi-events-block.o
  CC      qapi/qapi-events-char.o
  CC      qapi/qapi-events-common.o
  CC      qapi/qapi-events-crypto.o
  CC      qapi/qapi-events-introspect.o
  CC      qapi/qapi-events-job.o
  CC      qapi/qapi-events-migration.o
  CC      qapi/qapi-events-misc.o
  CC      qapi/qapi-events-net.o
  CC      qapi/qapi-events-rocker.o
  CC      qapi/qapi-events-run-state.o
  CC      qapi/qapi-events-sockets.o
  CC      qapi/qapi-events-tpm.o
  CC      qapi/qapi-events-trace.o
  CC      qapi/qapi-events-transaction.o
  CC      qapi/qapi-events-ui.o
  CC      qapi/qapi-introspect.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qnum.o
  CC      qobject/qstring.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qbool.o
  CC      qobject/qlit.o
  CC      qobject/qjson.o
  CC      qobject/qobject.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  CC      qobject/block-qdict.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/bufferiszero.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/async.o
  CC      util/aio-wait.o
  CC      util/thread-pool.o
  CC      util/qemu-timer.o
  CC      util/main-loop.o
  CC      util/iohandler.o
  CC      util/aio-posix.o
  CC      util/compatfd.o
  CC      util/event_notifier-posix.o
  CC      util/mmap-alloc.o
  CC      util/oslib-posix.o
  CC      util/qemu-openpty.o
  CC      util/qemu-thread-posix.o
  CC      util/memfd.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/module.o
  CC      util/host-utils.o
  CC      util/bitmap.o
  CC      util/bitops.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/cacheinfo.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/uri.o
  CC      util/notify.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/keyval.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/coroutine-ucontext.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/log.o
  CC      util/base64.o
  CC      util/pagesize.o
  CC      util/qdist.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      util/iova-tree.o
  CC      util/vfio-helpers.o
  CC      accel/kvm/trace.o
  CC      trace-root.o
  CC      accel/tcg/trace.o
  CC      audio/trace.o
  CC      block/trace.o
  CC      chardev/trace.o
  CC      crypto/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/acpi/trace.o
  CC      hw/alpha/trace.o
  CC      hw/arm/trace.o
  CC      hw/audio/trace.o
  CC      hw/block/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/display/trace.o
  CC      hw/dma/trace.o
  CC      hw/hppa/trace.o
  CC      hw/i2c/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/i386/trace.o
  CC      hw/ide/trace.o
  CC      hw/input/trace.o
  CC      hw/intc/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/misc/trace.o
  CC      hw/misc/macio/trace.o
  CC      hw/net/trace.o
  CC      hw/nvram/trace.o
  CC      hw/pci/trace.o
  CC      hw/pci-host/trace.o
  CC      hw/ppc/trace.o
  CC      hw/rdma/trace.o
  CC      hw/rdma/vmw/trace.o
  CC      hw/s390x/trace.o
  CC      hw/scsi/trace.o
  CC      hw/sd/trace.o
  CC      hw/sparc/trace.o
  CC      hw/sparc64/trace.o
  CC      hw/timer/trace.o
  CC      hw/tpm/trace.o
  CC      hw/usb/trace.o
  CC      hw/vfio/trace.o
  CC      hw/virtio/trace.o
  CC      hw/xen/trace.o
  CC      io/trace.o
  CC      linux-user/trace.o
  CC      migration/trace.o
  CC      nbd/trace.o
  CC      net/trace.o
  CC      qapi/trace.o
  CC      qom/trace.o
  CC      scsi/trace.o
  CC      target/arm/trace.o
  CC      target/i386/trace.o
  CC      target/mips/trace.o
  CC      target/ppc/trace.o
  CC      target/s390x/trace.o
  CC      target/sparc/trace.o
  CC      ui/trace.o
  CC      util/trace.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/clock-warp.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/gdbstub.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/linux-aio.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/replay.o
  CC      stubs/runstate-check.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/tpm.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/qmp_memory_device.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      stubs/pci-host-piix.o
  CC      stubs/ram-block.o
  CC      qemu-keymap.o
  CC      ui/input-keymap.o
  CC      contrib/ivshmem-client/ivshmem-client.o
  CC      contrib/ivshmem-client/main.o
  CC      contrib/ivshmem-server/ivshmem-server.o
  CC      contrib/ivshmem-server/main.o
  CC      qemu-nbd.o
  CC      block.o
  CC      blockjob.o
  CC      job.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkdebug.o
  CC      block/blkverify.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-posix.o
  CC      block/linux-aio.o
  CC      block/null.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/create.o
  CC      block/throttle-groups.o
  CC      block/nvme.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/iscsi-opts.o
  CC      block/accounting.o
  CC      block/dirty-bitmap.o
  CC      block/write-threshold.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/throttle.o
  CC      block/copy-on-read.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      scsi/utils.o
  CC      scsi/pr-manager.o
  CC      scsi/pr-manager-helper.o
  CC      block/iscsi.o
  CC      block/nfs.o
  CC      block/curl.o
  CC      block/rbd.o
  CC      block/gluster.o
  CC      block/ssh.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/hmac.o
  CC      crypto/hmac-nettle.o
  CC      crypto/aes.o
  CC      crypto/desrfb.o
  CC      crypto/cipher.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      crypto/block-luks.o
  CC      io/channel.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-watch.o
  CC      io/channel-websock.o
  CC      io/channel-util.o
  CC      io/dns-resolver.o
  CC      io/net-listener.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  GEN     qemu-img-cmds.h
  CC      qemu-io.o
  CC      fsdev/virtfs-proxy-helper.o
  CC      fsdev/9p-marshal.o
  CC      fsdev/9p-iov-marshal.o
  CC      scsi/qemu-pr-helper.o
  CC      qemu-bridge-helper.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      job-qmp.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-posix.o
  CC      bt-host.o
  CC      dma-helpers.o
  CC      bt-vhci.o
  CC      vl.o
  CC      tpm.o
  CC      qemu-seccomp.o
  CC      device_tree.o
  CC      qapi/qapi-commands.o
  CC      qapi/qapi-commands-block-core.o
  CC      qapi/qapi-commands-block.o
  CC      qapi/qapi-commands-char.o
  CC      qapi/qapi-commands-common.o
  CC      qapi/qapi-commands-crypto.o
  CC      qapi/qapi-commands-introspect.o
  CC      qapi/qapi-commands-job.o
  CC      qapi/qapi-commands-migration.o
  CC      qapi/qapi-commands-misc.o
  CC      qapi/qapi-commands-net.o
  CC      qapi/qapi-commands-rocker.o
  CC      qapi/qapi-commands-run-state.o
  CC      qapi/qapi-commands-sockets.o
  CC      qapi/qapi-commands-tpm.o
  CC      qapi/qapi-commands-trace.o
  CC      qapi/qapi-commands-transaction.o
  CC      qapi/qapi-commands-ui.o
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/wavcapture.o
  CC      backends/rng.o
  CC      backends/rng-egd.o
  CC      backends/rng-random.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/hostmem-file.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      backends/cryptodev-vhost.o
  CC      backends/cryptodev-vhost-user.o
  CC      backends/hostmem-memfd.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      chardev/baum.o
  CC      disas/alpha.o
  CC      disas/arm.o
  CXX     disas/arm-a64.o
  CC      disas/cris.o
  CC      disas/hppa.o
  CC      disas/i386.o
  CC      disas/m68k.o
  CC      disas/microblaze.o
  CC      disas/mips.o
  CC      disas/nios2.o
  CC      disas/moxie.o
  CC      disas/ppc.o
  CC      disas/riscv.o
  CC      disas/s390.o
  CC      disas/sh4.o
  CC      disas/sparc.o
  CC      disas/lm32.o
  CC      disas/xtensa.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CC      fsdev/qemu-fsdev.o
  CC      fsdev/qemu-fsdev-opts.o
  CC      fsdev/qemu-fsdev-throttle.o
  CC      fsdev/qemu-fsdev-dummy.o
  CC      hw/9pfs/9p.o
  CC      hw/9pfs/9p-util.o
  CC      hw/9pfs/9p-local.o
  CC      hw/9pfs/9p-xattr.o
  CC      hw/9pfs/9p-xattr-user.o
  CC      hw/9pfs/9p-posix-acl.o
  CC      hw/9pfs/coth.o
  CC      hw/9pfs/cofs.o
  CC      hw/9pfs/codir.o
  CC      hw/9pfs/cofile.o
  CC      hw/9pfs/coxattr.o
  CC      hw/9pfs/9p-synth.o
  CC      hw/9pfs/9p-handle.o
  CC      hw/9pfs/9p-proxy.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/cs4231.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/milkymist-ac97.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/core.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/escc.o
  CC      hw/char/parallel.o
  CC      hw/char/parallel-isa.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/xilinx_uartlite.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/etraxfs_ser.o
  CC      hw/char/debugcon.o
  CC      hw/char/grlib_apbuart.o
  CC      hw/char/imx_serial.o
  CC      hw/char/lm32_juart.o
  CC      hw/char/lm32_uart.o
  CC      hw/char/milkymist-uart.o
  CC      hw/char/sclpconsole.o
  CC      hw/char/sclpconsole-lm.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/qdev-fw.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/empty_slot.o
  CC      hw/core/stream.o
  CC      hw/core/ptimer.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/loader.o
  CC      hw/core/loader-fit.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/split-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/cpu/core.o
  CC      hw/display/ramfb.o
  CC      hw/display/ramfb-standalone.o
  CC      hw/display/ads7846.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/g364fb.o
  CC      hw/display/jazz_led.o
  CC      hw/display/pl110.o
  CC      hw/display/sii9022.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/bochs-display.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vga-isa-mm.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/milkymist-vgafb.o
  CC      hw/display/tc6393xb.o
  CC      hw/display/milkymist-tmu2.o
  CC      hw/dma/puv3_dma.o
  CC      hw/dma/rc4030.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i82374.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xilinx_axidma.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/dma/etraxfs_dma.o
  CC      hw/dma/sparc32_dma.o
  CC      hw/dma/xlnx-zdma.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/puv3_gpio.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/mpc8xxx.o
  CC      hw/gpio/gpio_key.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/cmd646.o
  CC      hw/ide/macio.o
  CC      hw/ide/mmio.o
  CC      hw/ide/via.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/ide/ich.o
  CC      hw/ide/ahci-allwinner.o
  CC      hw/ide/sii3112.o
  CC      hw/input/adb.o
  CC      hw/input/adb-mouse.o
  CC      hw/input/adb-kbd.o
  CC      hw/input/hid.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/input/virtio-input-host.o
  CC      hw/intc/heathrow_pic.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/puv3_intc.o
  CC      hw/intc/xilinx_intc.o
  CC      hw/intc/xlnx-pmu-iomod-intc.o
  CC      hw/intc/xlnx-zynqmp-ipi.o
  CC      hw/intc/etraxfs_pic.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/imx_gpcv2.o
  CC      hw/intc/lm32_pic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/slavio_intctl.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/openpic.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/isa-superio.o
  CC      hw/isa/smc37c669-superio.o
  CC      hw/isa/apm.o
  CC      hw/isa/i82378.o
  CC      hw/isa/pc87312.o
  CC      hw/isa/piix4.o
  CC      hw/isa/vt82c686.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/memory-device.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/pca9552.o
  CC      hw/misc/unimp.o
  CC      hw/misc/vmcoreinfo.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/misc/mos6522.o
  CC      hw/misc/puv3_pm.o
  CC      hw/misc/macio/macio.o
  CC      hw/misc/macio/cuda.o
  CC      hw/misc/macio/pmu.o
  CC      hw/misc/macio/mac_dbdma.o
  CC      hw/misc/macio/gpio.o
  CC      hw/net/dp8393x.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/opencores_eth.o
  CC      hw/net/xgmac.o
  CC      hw/net/mipsnet.o
  CC      hw/net/xilinx_axienet.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/lance.o
  CC      hw/net/sunhme.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/sungem.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/net/can/can_sja1000.o
  CC      hw/net/can/can_kvaser_pci.o
  CC      hw/net/can/can_pcm3680_pci.o
  CC      hw/net/can/can_mioe3680_pci.o
  CC      hw/nvram/ds1225y.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/eeprom_at24c.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/nvram/mac_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/pcie_pci_bridge.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-bridge/dec.o
  CC      hw/pci-bridge/simba.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/prep.o
  CC      hw/pci-host/grackle.o
  CC      hw/pci-host/uninorth.o
  CC      hw/pci-host/ppce500.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/sabre.o
  CC      hw/pci-host/bonito.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci-host/xilinx-pcie.o
  CC      hw/pci-host/designware.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/pl181.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdmmc-internal.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/ssi/mss-spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/i8254.o
  CC      hw/timer/m48t59.o
  CC      hw/timer/m48t59-isa.o
  CC      hw/timer/pl031.o
  CC      hw/timer/puv3_ost.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/xilinx_timer.o
  CC      hw/timer/slavio_timer.o
  CC      hw/timer/etraxfs_timer.o
  CC      hw/timer/grlib_gptimer.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/lm32_timer.o
  CC      hw/timer/milkymist-sysctl.o
  CC      hw/timer/xlnx-zynqmp-rtc.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/sun4v-rtc.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/timer/mss-timer.o
  CC      hw/tpm/tpm_util.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/tpm/tpm_crb.o
  CC      hw/tpm/tpm_emulator.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/ccid-card-passthru.o
  CC      hw/usb/ccid-card-emulated.o
  CC      hw/usb/dev-mtp.o
  CC      hw/usb/redirect.o
  CC      hw/usb/quirks.o
  CC      hw/usb/host-libusb.o
  CC      hw/usb/host-stub.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_diag288.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      migration/migration.o
  CC      migration/socket.o
  CC      migration/fd.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/savevm.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate.o
  CC      migration/vmstate-types.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/global_state.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/block-dirty-bitmap.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/l2tpv3.o
  CC      net/vhost-user.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/filter-mirror.o
  CC      net/colo-compare.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap.o
  CC      net/tap-linux.o
  CC      net/can/can_core.o
  CC      net/can/can_host.o
  CC      net/can/can_socketcan.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-char.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/if.o
  CC      slirp/ip_icmp.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_input.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/misc.o
  CC      slirp/sbuf.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-legacy.o
  CC      ui/input-linux.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-auth-sasl.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  VERT    ui/shader/texture-blit-vert.h
  VERT    ui/shader/texture-blit-flip-vert.h
  CC      ui/console-gl.o
  FRAG    ui/shader/texture-blit-frag.h
  CC      ui/egl-helpers.o
  CC      ui/egl-context.o
  CC      ui/egl-headless.o
  CC      audio/ossaudio.o
  CC      ui/sdl2.o
  CC      ui/sdl2-input.o
  CC      ui/sdl2-2d.o
  CC      ui/sdl2-gl.o
  CC      ui/x_keymap.o
  CC      ui/gtk.o
  CC      ui/gtk-egl.o
  CC      ui/gtk-gl-area.o
  CC      ui/curses.o
  CC      chardev/char.o
  CC      chardev/char-fd.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-parallel.o
  CC      chardev/char-pipe.o
  CC      chardev/char-pty.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  CCAS    s390-ccw/start.o
  LINK    tests/qemu-iotests/socket_scm_helper
  CC      s390-ccw/main.o
  GEN     qemu-doc.html
  GEN     qemu-doc.txt
  GEN     qemu.1
  GEN     docs/interop/qemu-qmp-ref.html
  GEN     docs/interop/qemu-qmp-ref.txt
  GEN     docs/interop/qemu-qmp-ref.7
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      s390-ccw/bootmap.o
  CC      qga/main.o
  CC      qga/commands-posix.o
  CC      qga/channel-posix.o
  CC      s390-ccw/jump2ipl.o
  CC      s390-ccw/sclp.o
  CC      s390-ccw/menu.o
  CC      s390-ccw/virtio.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      s390-ccw/virtio-scsi.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  CC      qga/qapi-generated/qga-qapi-commands.o
  CC      s390-ccw/virtio-blkdev.o
  AR      libqemuutil.a
  CC      qemu-img.o
  CC      s390-ccw/libc.o
  LINK    qemu-io
s390-netboot.img not built since roms/SLOF/ is not available.
  BUILD   s390-ccw/s390-ccw.elf
  STRIP   s390-ccw/s390-ccw.img
  LINK    fsdev/virtfs-proxy-helper
  LINK    scsi/qemu-pr-helper
  LINK    qemu-bridge-helper
  CC      ui/shader.o
  GEN     docs/interop/qemu-ga-ref.html
  GEN     docs/interop/qemu-ga-ref.txt
  GEN     docs/interop/qemu-ga-ref.7
  LINK    qemu-ga
  LINK    qemu-keymap
  LINK    ivshmem-client
  LINK    ivshmem-server
  LINK    qemu-nbd
  LINK    qemu-img
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     aarch64-softmmu/config-target.h
  CC      aarch64-softmmu/exec.o
  GEN     alpha-softmmu/hmp-commands.h
  GEN     alpha-softmmu/hmp-commands-info.h
  GEN     alpha-softmmu/config-target.h
  GEN     cris-softmmu/hmp-commands.h
  CC      alpha-softmmu/exec.o
  GEN     arm-softmmu/hmp-commands.h
  GEN     cris-softmmu/hmp-commands-info.h
  GEN     cris-softmmu/config-target.h
  GEN     arm-softmmu/hmp-commands-info.h
  CC      cris-softmmu/exec.o
  GEN     arm-softmmu/config-target.h
  CC      arm-softmmu/exec.o
  CC      alpha-softmmu/tcg/tcg.o
  CC      arm-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      cris-softmmu/tcg/tcg.o
  CC      alpha-softmmu/tcg/tcg-op.o
  CC      arm-softmmu/tcg/tcg-op.o
  CC      cris-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      alpha-softmmu/tcg/tcg-op-vec.o
  CC      alpha-softmmu/tcg/tcg-op-gvec.o
  CC      cris-softmmu/tcg/tcg-op-vec.o
  CC      arm-softmmu/tcg/tcg-op-vec.o
  CC      aarch64-softmmu/tcg/tcg-op-vec.o
  CC      cris-softmmu/tcg/tcg-op-gvec.o
  CC      arm-softmmu/tcg/tcg-op-gvec.o
  CC      alpha-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/tcg/tcg-op-gvec.o
  CC      alpha-softmmu/tcg/optimize.o
  CC      cris-softmmu/tcg/tcg-common.o
  CC      arm-softmmu/tcg/tcg-common.o
  CC      cris-softmmu/tcg/optimize.o
  CC      alpha-softmmu/fpu/softfloat.o
  CC      arm-softmmu/tcg/optimize.o
  CC      cris-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      arm-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      alpha-softmmu/disas.o
  CC      cris-softmmu/disas.o
  CC      alpha-softmmu/arch_init.o
  CC      cris-softmmu/arch_init.o
  CC      alpha-softmmu/cpus.o
  CC      cris-softmmu/cpus.o
  CC      arm-softmmu/disas.o
  CC      cris-softmmu/monitor.o
  CC      alpha-softmmu/monitor.o
  CC      aarch64-softmmu/disas.o
  GEN     arm-softmmu/gdbstub-xml.c
  CC      arm-softmmu/arch_init.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      arm-softmmu/cpus.o
  CC      cris-softmmu/gdbstub.o
  CC      arm-softmmu/monitor.o
  CC      alpha-softmmu/gdbstub.o
  CC      cris-softmmu/balloon.o
  CC      aarch64-softmmu/arch_init.o
  CC      cris-softmmu/ioport.o
  CC      alpha-softmmu/balloon.o
  CC      aarch64-softmmu/cpus.o
  CC      cris-softmmu/numa.o
  CC      alpha-softmmu/ioport.o
  CC      arm-softmmu/gdbstub.o
  CC      cris-softmmu/qtest.o
  CC      aarch64-softmmu/monitor.o
  CC      alpha-softmmu/numa.o
  CC      cris-softmmu/memory.o
  CC      arm-softmmu/balloon.o
  CC      alpha-softmmu/qtest.o
  CC      arm-softmmu/ioport.o
  CC      alpha-softmmu/memory.o
  CC      aarch64-softmmu/gdbstub.o
  CC      arm-softmmu/numa.o
  CC      cris-softmmu/memory_mapping.o
  CC      cris-softmmu/dump.o
  CC      arm-softmmu/qtest.o
  CC      aarch64-softmmu/balloon.o
  CC      arm-softmmu/memory.o
  CC      alpha-softmmu/memory_mapping.o
  CC      aarch64-softmmu/ioport.o
  CC      cris-softmmu/migration/ram.o
  CC      alpha-softmmu/dump.o
  CC      aarch64-softmmu/numa.o
  CC      alpha-softmmu/migration/ram.o
  CC      arm-softmmu/memory_mapping.o
  CC      cris-softmmu/accel/accel.o
  CC      aarch64-softmmu/qtest.o
  CC      arm-softmmu/dump.o
  CC      cris-softmmu/accel/stubs/hax-stub.o
  CC      aarch64-softmmu/memory.o
  CC      cris-softmmu/accel/stubs/hvf-stub.o
  CC      alpha-softmmu/accel/accel.o
  CC      arm-softmmu/migration/ram.o
  CC      cris-softmmu/accel/stubs/whpx-stub.o
  CC      alpha-softmmu/accel/stubs/hax-stub.o
  CC      cris-softmmu/accel/stubs/kvm-stub.o
  CC      cris-softmmu/accel/tcg/tcg-all.o
  CC      alpha-softmmu/accel/stubs/hvf-stub.o
  CC      cris-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      alpha-softmmu/accel/stubs/whpx-stub.o
  CC      arm-softmmu/accel/accel.o
  CC      alpha-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/dump.o
  CC      arm-softmmu/accel/stubs/hax-stub.o
  CC      alpha-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/accel/stubs/hvf-stub.o
  CC      arm-softmmu/accel/stubs/whpx-stub.o
  CC      alpha-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/migration/ram.o
  CC      arm-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/accel/accel.o
  CC      cris-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64-softmmu/accel/stubs/hax-stub.o
  CC      cris-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      aarch64-softmmu/accel/stubs/hvf-stub.o
  CC      aarch64-softmmu/accel/stubs/whpx-stub.o
  CC      alpha-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/accel/tcg/tcg-all.o
  CC      alpha-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      cris-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/accel/tcg/cpu-exec-common.o
  CC      alpha-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/accel/tcg/tcg-runtime.o
  CC      cris-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      alpha-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/accel/tcg/translator.o
  CC      alpha-softmmu/accel/tcg/translate-all.o
  CC      cris-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/accel/tcg/cpu-exec.o
  CC      alpha-softmmu/accel/tcg/translator.o
  CC      arm-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/hw/core/null-machine.o
  CC      arm-softmmu/accel/tcg/translate-all.o
  CC      cris-softmmu/hw/misc/mmio_interface.o
  CC      alpha-softmmu/hw/9pfs/virtio-9p-device.o
  CC      aarch64-softmmu/accel/tcg/tcg-runtime.o
  CC      cris-softmmu/hw/net/etraxfs_eth.o
  CC      alpha-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      cris-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/accel/tcg/translator.o
  CC      cris-softmmu/hw/net/rocker/qmp-norocker.o
  CC      alpha-softmmu/hw/block/vhost-user-blk.o
  CC      cris-softmmu/hw/vfio/common.o
  CC      arm-softmmu/hw/9pfs/virtio-9p-device.o
  CC      alpha-softmmu/hw/block/dataplane/virtio-blk.o
  CC      arm-softmmu/hw/adc/stm32f2xx_adc.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec.o
  CC      alpha-softmmu/hw/char/virtio-serial-bus.o
  CC      cris-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec-common.o
  CC      alpha-softmmu/hw/core/generic-loader.o
  CC      cris-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/block/vhost-user-blk.o
  CC      aarch64-softmmu/accel/tcg/translate-all.o
  CC      alpha-softmmu/hw/core/null-machine.o
  CC      alpha-softmmu/hw/display/vga.o
  CC      arm-softmmu/hw/block/dataplane/virtio-blk.o
  CC      cris-softmmu/hw/cris/boot.o
  CC      arm-softmmu/hw/char/exynos4210_uart.o
  CC      aarch64-softmmu/accel/tcg/translator.o
  CC      cris-softmmu/hw/cris/axis_dev88.o
  CC      arm-softmmu/hw/char/omap_uart.o
  CC      arm-softmmu/hw/char/digic-uart.o
  CC      alpha-softmmu/hw/display/virtio-gpu.o
  CC      cris-softmmu/target/cris/translate.o
  CC      arm-softmmu/hw/char/stm32f2xx_usart.o
  CC      aarch64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      arm-softmmu/hw/char/bcm2835_aux.o
  CC      alpha-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      arm-softmmu/hw/char/virtio-serial-bus.o
  CC      alpha-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      alpha-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/core/generic-loader.o
  CC      alpha-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/block/vhost-user-blk.o
  CC      arm-softmmu/hw/core/null-machine.o
  CC      cris-softmmu/target/cris/op_helper.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      arm-softmmu/hw/cpu/arm11mpcore.o
  CC      alpha-softmmu/hw/net/virtio-net.o
  CC      cris-softmmu/target/cris/helper.o
  CC      arm-softmmu/hw/cpu/realview_mpcore.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      arm-softmmu/hw/cpu/a9mpcore.o
  CC      cris-softmmu/target/cris/cpu.o
  CC      alpha-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      cris-softmmu/target/cris/gdbstub.o
  CC      arm-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi.o
  CC      cris-softmmu/target/cris/mmu.o
  CC      arm-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      cris-softmmu/target/cris/machine.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/trace/control-target.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      arm-softmmu/hw/display/omap_lcdc.o
  CC      cris-softmmu/trace/generated-helpers.o
  CC      alpha-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/display/pxa2xx_lcd.o
  CC      aarch64-softmmu/hw/core/null-machine.o
  LINK    cris-softmmu/qemu-system-cris
  CC      alpha-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      alpha-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      alpha-softmmu/hw/timer/mc146818rtc.o
  CC      arm-softmmu/hw/display/bcm2835_fb.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/display/omap_dss.o
  CC      arm-softmmu/hw/display/vga.o
  CC      alpha-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      arm-softmmu/hw/display/virtio-gpu.o
  CC      alpha-softmmu/hw/vfio/pci.o
  GEN     hppa-softmmu/hmp-commands.h
  GEN     hppa-softmmu/hmp-commands-info.h
  GEN     hppa-softmmu/config-target.h
  CC      hppa-softmmu/exec.o
  CC      arm-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      alpha-softmmu/hw/vfio/pci-quirks.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      arm-softmmu/hw/display/virtio-gpu-pci.o
  CC      alpha-softmmu/hw/vfio/display.o
  CC      arm-softmmu/hw/dma/omap_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      hppa-softmmu/tcg/tcg.o
  CC      alpha-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/dma/soc_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      alpha-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/dma/pxa2xx_dma.o
  CC      alpha-softmmu/hw/virtio/virtio.o
  CC      arm-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-softmmu/hw/gpio/omap_gpio.o
  CC      hppa-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      arm-softmmu/hw/gpio/imx_gpio.o
  CC      alpha-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      arm-softmmu/hw/gpio/bcm2835_gpio.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto.o
  CC      arm-softmmu/hw/i2c/omap_i2c.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      arm-softmmu/hw/input/pxa2xx_keypad.o
  CC      alpha-softmmu/hw/virtio/vhost.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      arm-softmmu/hw/input/tsc210x.o
  CC      alpha-softmmu/hw/virtio/vhost-backend.o
  CC      hppa-softmmu/tcg/tcg-op-vec.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      alpha-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      arm-softmmu/hw/intc/armv7m_nvic.o
  CC      hppa-softmmu/tcg/tcg-op-gvec.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      alpha-softmmu/hw/virtio/vhost-vsock.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      arm-softmmu/hw/intc/exynos4210_gic.o
  CC      alpha-softmmu/hw/alpha/dp264.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      alpha-softmmu/hw/alpha/pci.o
  CC      arm-softmmu/hw/intc/exynos4210_combiner.o
  CC      alpha-softmmu/hw/alpha/typhoon.o
  CC      hppa-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
  CC      arm-softmmu/hw/intc/omap_intc.o
  CC      hppa-softmmu/tcg/optimize.o
  CC      alpha-softmmu/target/alpha/machine.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      arm-softmmu/hw/intc/bcm2835_ic.o
  CC      alpha-softmmu/target/alpha/translate.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      arm-softmmu/hw/intc/bcm2836_control.o
  CC      hppa-softmmu/fpu/softfloat.o
  CC      arm-softmmu/hw/intc/allwinner-a10-pic.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      arm-softmmu/hw/intc/aspeed_vic.o
  CC      arm-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      alpha-softmmu/target/alpha/helper.o
  CC      alpha-softmmu/target/alpha/cpu.o
  CC      arm-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      alpha-softmmu/target/alpha/int_helper.o
  CC      arm-softmmu/hw/misc/arm_sysctl.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      arm-softmmu/hw/misc/cbus.o
  CC      alpha-softmmu/target/alpha/fpu_helper.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      arm-softmmu/hw/misc/exynos4210_pmu.o
  CC      alpha-softmmu/target/alpha/vax_helper.o
  CC      arm-softmmu/hw/misc/exynos4210_clk.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      arm-softmmu/hw/misc/exynos4210_rng.o
  CC      alpha-softmmu/target/alpha/sys_helper.o
  CC      hppa-softmmu/disas.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      arm-softmmu/hw/misc/imx_ccm.o
  CC      alpha-softmmu/target/alpha/mem_helper.o
  CC      hppa-softmmu/arch_init.o
  CC      arm-softmmu/hw/misc/imx31_ccm.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      alpha-softmmu/target/alpha/gdbstub.o
  CC      hppa-softmmu/cpus.o
  CC      arm-softmmu/hw/misc/imx25_ccm.o
  GEN     trace/generated-helpers.c
  CC      alpha-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      alpha-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/misc/imx6_ccm.o
  CC      hppa-softmmu/monitor.o
  CC      arm-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  LINK    alpha-softmmu/qemu-system-alpha
  CC      arm-softmmu/hw/misc/imx7_ccm.o
  CC      aarch64-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/misc/imx2_wdt.o
  CC      arm-softmmu/hw/misc/imx7_snvs.o
  CC      hppa-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      arm-softmmu/hw/misc/imx7_gpr.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      arm-softmmu/hw/misc/mst_fpga.o
  CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
  CC      arm-softmmu/hw/misc/omap_clk.o
  CC      arm-softmmu/hw/misc/omap_gpmc.o
  GEN     i386-softmmu/hmp-commands.h
  GEN     i386-softmmu/hmp-commands-info.h
  GEN     i386-softmmu/config-target.h
  CC      i386-softmmu/exec.o
  CC      arm-softmmu/hw/misc/omap_l4.o
  CC      arm-softmmu/hw/misc/omap_sdrc.o
  CC      arm-softmmu/hw/misc/omap_tap.o
  CC      arm-softmmu/hw/misc/bcm2835_mbox.o
  CC      arm-softmmu/hw/misc/bcm2835_property.o
  CC      arm-softmmu/hw/misc/bcm2835_rng.o
  CC      arm-softmmu/hw/misc/zynq_slcr.o
  CC      arm-softmmu/hw/misc/zynq-xadc.o
  CC      i386-softmmu/tcg/tcg.o
  CC      arm-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      arm-softmmu/hw/misc/mps2-fpgaio.o
  CC      arm-softmmu/hw/misc/mps2-scc.o
  CC      arm-softmmu/hw/misc/tz-mpc.o
  CC      arm-softmmu/hw/misc/tz-ppc.o
  CC      arm-softmmu/hw/misc/iotkit-secctl.o
  CC      arm-softmmu/hw/misc/aspeed_scu.o
  CC      arm-softmmu/hw/misc/aspeed_sdmc.o
  CC      i386-softmmu/tcg/tcg-op.o
  CC      arm-softmmu/hw/misc/mmio_interface.o
  CC      arm-softmmu/hw/misc/msf2-sysreg.o
  CC      arm-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/hw/pcmcia/pxa2xx.o
  CC      arm-softmmu/hw/scsi/virtio-scsi.o
  CC      arm-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      i386-softmmu/tcg/tcg-op-vec.o
  CC      arm-softmmu/hw/scsi/vhost-scsi-common.o
  CC      arm-softmmu/hw/scsi/vhost-scsi.o
  CC      i386-softmmu/tcg/tcg-op-gvec.o
  CC      arm-softmmu/hw/scsi/vhost-user-scsi.o
  CC      arm-softmmu/hw/sd/omap_mmc.o
  CC      arm-softmmu/hw/sd/pxa2xx_mmci.o
  CC      arm-softmmu/hw/sd/bcm2835_sdhost.o
  CC      arm-softmmu/hw/ssi/omap_spi.o
  CC      arm-softmmu/hw/ssi/imx_spi.o
  CC      i386-softmmu/tcg/tcg-common.o
  CC      arm-softmmu/hw/timer/exynos4210_mct.o
  CC      i386-softmmu/tcg/optimize.o
  CC      arm-softmmu/hw/timer/exynos4210_pwm.o
  CC      arm-softmmu/hw/timer/exynos4210_rtc.o
  CC      arm-softmmu/hw/timer/omap_gptimer.o
  CC      i386-softmmu/fpu/softfloat.o
  CC      arm-softmmu/hw/timer/omap_synctimer.o
  CC      arm-softmmu/hw/timer/pxa2xx_timer.o
  CC      arm-softmmu/hw/timer/digic-timer.o
  CC      arm-softmmu/hw/timer/allwinner-a10-pit.o
  CC      arm-softmmu/hw/usb/tusb6010.o
  CC      arm-softmmu/hw/usb/chipidea.o
  CC      arm-softmmu/hw/vfio/common.o
  CC      arm-softmmu/hw/vfio/pci.o
  CC      arm-softmmu/hw/vfio/pci-quirks.o
  CC      i386-softmmu/disas.o
  GEN     i386-softmmu/gdbstub-xml.c
  CC      i386-softmmu/arch_init.o
  CC      i386-softmmu/cpus.o
  CC      arm-softmmu/hw/vfio/display.o
  CC      arm-softmmu/hw/vfio/platform.o
  CC      i386-softmmu/monitor.o
  CC      arm-softmmu/hw/vfio/calxeda-xgmac.o
  CC      arm-softmmu/hw/vfio/amd-xgbe.o
  CC      arm-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/virtio/virtio.o
  CC      i386-softmmu/gdbstub.o
  CC      i386-softmmu/balloon.o
  CC      i386-softmmu/ioport.o
  CC      arm-softmmu/hw/virtio/virtio-balloon.o
  CC      i386-softmmu/numa.o
  CC      arm-softmmu/hw/virtio/virtio-crypto.o
  CC      i386-softmmu/qtest.o
  CC      arm-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      i386-softmmu/memory.o
  CC      arm-softmmu/hw/virtio/vhost.o
  CC      arm-softmmu/hw/virtio/vhost-backend.o
  CC      arm-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/memory_mapping.o
  CC      i386-softmmu/dump.o
  CC      arm-softmmu/hw/virtio/vhost-vsock.o
  CC      arm-softmmu/hw/arm/boot.o
  CC      i386-softmmu/migration/ram.o
  CC      arm-softmmu/hw/arm/virt.o
  CC      arm-softmmu/hw/arm/sysbus-fdt.o
  CC      i386-softmmu/accel/accel.o
  CC      i386-softmmu/accel/stubs/hax-stub.o
  CC      arm-softmmu/hw/arm/virt-acpi-build.o
  CC      i386-softmmu/accel/stubs/hvf-stub.o
  CC      i386-softmmu/accel/stubs/whpx-stub.o
  CC      i386-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/hw/arm/digic_boards.o
  CC      i386-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/hw/arm/exynos4_boards.o
  CC      i386-softmmu/accel/tcg/cputlb.o
  CC      arm-softmmu/hw/arm/highbank.o
  CC      arm-softmmu/hw/arm/integratorcp.o
  CC      arm-softmmu/hw/arm/mainstone.o
  CC      arm-softmmu/hw/arm/musicpal.o
  CC      arm-softmmu/hw/arm/netduino2.o
  CC      arm-softmmu/hw/arm/nseries.o
  CC      arm-softmmu/hw/arm/omap_sx1.o
  CC      arm-softmmu/hw/arm/palm.o
  CC      i386-softmmu/accel/tcg/tcg-runtime.o
  CC      i386-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      arm-softmmu/hw/arm/gumstix.o
  CC      arm-softmmu/hw/arm/spitz.o
  CC      arm-softmmu/hw/arm/tosa.o
  CC      i386-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/hw/arm/z2.o
  CC      i386-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/hw/arm/realview.o
  CC      i386-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/hw/arm/stellaris.o
  CC      i386-softmmu/accel/tcg/translator.o
  CC      arm-softmmu/hw/arm/collie.o
  CC      arm-softmmu/hw/arm/vexpress.o
  CC      i386-softmmu/hw/9pfs/virtio-9p-device.o
  CC      arm-softmmu/hw/arm/versatilepb.o
  CC      i386-softmmu/hw/block/virtio-blk.o
  CC      arm-softmmu/hw/arm/xilinx_zynq.o
  CC      i386-softmmu/hw/block/vhost-user-blk.o
  CC      arm-softmmu/hw/arm/armv7m.o
  CC      i386-softmmu/hw/block/dataplane/virtio-blk.o
  CC      arm-softmmu/hw/arm/exynos4210.o
  CC      i386-softmmu/hw/char/virtio-serial-bus.o
  CC      arm-softmmu/hw/arm/pxa2xx.o
  CC      i386-softmmu/hw/core/generic-loader.o
  CC      i386-softmmu/hw/core/null-machine.o
  CC      i386-softmmu/hw/display/vga.o
  CC      arm-softmmu/hw/arm/pxa2xx_gpio.o
  CC      arm-softmmu/hw/arm/pxa2xx_pic.o
  CC      arm-softmmu/hw/arm/digic.o
  CC      arm-softmmu/hw/arm/omap1.o
  CC      i386-softmmu/hw/display/virtio-gpu.o
  CC      i386-softmmu/hw/display/virtio-gpu-3d.o
  CC      arm-softmmu/hw/arm/omap2.o
  CC      i386-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-softmmu/hw/arm/strongarm.o
  CC      i386-softmmu/hw/display/virtio-vga.o
  CC      arm-softmmu/hw/arm/allwinner-a10.o
  CC      i386-softmmu/hw/intc/apic.o
  CC      arm-softmmu/hw/arm/cubieboard.o
  CC      arm-softmmu/hw/arm/bcm2835_peripherals.o
  CC      i386-softmmu/hw/intc/apic_common.o
  CC      i386-softmmu/hw/intc/ioapic.o
  CC      arm-softmmu/hw/arm/bcm2836.o
  CC      i386-softmmu/hw/isa/lpc_ich9.o
  CC      arm-softmmu/hw/arm/raspi.o
  CC      i386-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/arm/stm32f205_soc.o
  CC      arm-softmmu/hw/arm/fsl-imx25.o
  CC      i386-softmmu/hw/misc/pvpanic.o
  CC      arm-softmmu/hw/arm/imx25_pdk.o
  CC      i386-softmmu/hw/misc/mmio_interface.o
  CC      i386-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/arm/fsl-imx31.o
  CC      arm-softmmu/hw/arm/kzm.o
  CC      arm-softmmu/hw/arm/fsl-imx6.o
  CC      i386-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/hw/arm/sabrelite.o
  CC      i386-softmmu/hw/scsi/virtio-scsi.o
  CC      arm-softmmu/hw/arm/aspeed_soc.o
  CC      i386-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      arm-softmmu/hw/arm/aspeed.o
  CC      i386-softmmu/hw/scsi/vhost-scsi-common.o
  CC      arm-softmmu/hw/arm/mps2.o
  CC      i386-softmmu/hw/scsi/vhost-scsi.o
  CC      arm-softmmu/hw/arm/mps2-tz.o
  CC      i386-softmmu/hw/scsi/vhost-user-scsi.o
  CC      i386-softmmu/hw/timer/mc146818rtc.o
  CC      arm-softmmu/hw/arm/msf2-soc.o
  CC      arm-softmmu/hw/arm/msf2-som.o
  CC      i386-softmmu/hw/vfio/common.o
  CC      arm-softmmu/hw/arm/iotkit.o
  CC      arm-softmmu/hw/arm/fsl-imx7.o
  CC      i386-softmmu/hw/vfio/pci.o
  CC      arm-softmmu/hw/arm/mcimx7d-sabre.o
  CC      arm-softmmu/target/arm/arm-semi.o
  CC      arm-softmmu/target/arm/machine.o
  CC      arm-softmmu/target/arm/psci.o
  CC      i386-softmmu/hw/vfio/pci-quirks.o
  CC      arm-softmmu/target/arm/arch_dump.o
  CC      arm-softmmu/target/arm/monitor.o
  CC      arm-softmmu/target/arm/kvm-stub.o
  CC      i386-softmmu/hw/vfio/display.o
  CC      arm-softmmu/target/arm/translate.o
  CC      i386-softmmu/hw/vfio/platform.o
  CC      i386-softmmu/hw/vfio/spapr.o
  CC      i386-softmmu/hw/virtio/virtio.o
  CC      i386-softmmu/hw/virtio/virtio-balloon.o
  CC      i386-softmmu/hw/virtio/virtio-crypto.o
  CC      i386-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      i386-softmmu/hw/virtio/vhost.o
  CC      i386-softmmu/hw/virtio/vhost-backend.o
  CC      i386-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/hw/virtio/vhost-vsock.o
  CC      i386-softmmu/hw/i386/multiboot.o
  CC      i386-softmmu/hw/i386/pc.o
  CC      i386-softmmu/hw/i386/pc_piix.o
  CC      i386-softmmu/hw/i386/pc_q35.o
  CC      arm-softmmu/target/arm/op_helper.o
  CC      i386-softmmu/hw/i386/pc_sysfw.o
  CC      i386-softmmu/hw/i386/x86-iommu.o
  CC      i386-softmmu/hw/i386/intel_iommu.o
  CC      arm-softmmu/target/arm/helper.o
  CC      i386-softmmu/hw/i386/amd_iommu.o
  CC      i386-softmmu/hw/i386/vmport.o
  CC      i386-softmmu/hw/i386/vmmouse.o
  CC      i386-softmmu/hw/i386/kvmvapic.o
  CC      i386-softmmu/hw/i386/acpi-build.o
  CC      i386-softmmu/target/i386/helper.o
  CC      arm-softmmu/target/arm/cpu.o
  CC      i386-softmmu/target/i386/cpu.o
  CC      arm-softmmu/target/arm/neon_helper.o
  CC      i386-softmmu/target/i386/gdbstub.o
  CC      i386-softmmu/target/i386/xsave_helper.o
  CC      i386-softmmu/target/i386/translate.o
  CC      arm-softmmu/target/arm/iwmmxt_helper.o
  CC      arm-softmmu/target/arm/vec_helper.o
  CC      arm-softmmu/target/arm/gdbstub.o
  CC      arm-softmmu/target/arm/crypto_helper.o
  CC      arm-softmmu/target/arm/arm-powerctl.o
  GEN     trace/generated-helpers.c
  CC      arm-softmmu/trace/control-target.o
  CC      arm-softmmu/gdbstub-xml.o
  CC      arm-softmmu/trace/generated-helpers.o
  LINK    arm-softmmu/qemu-system-arm
  CC      hppa-softmmu/balloon.o
  CC      hppa-softmmu/ioport.o
  GEN     lm32-softmmu/hmp-commands.h
  GEN     lm32-softmmu/hmp-commands-info.h
  GEN     lm32-softmmu/config-target.h
  CC      lm32-softmmu/exec.o
  CC      hppa-softmmu/numa.o
  CC      i386-softmmu/target/i386/bpt_helper.o
  CC      hppa-softmmu/qtest.o
  CC      i386-softmmu/target/i386/cc_helper.o
  CC      hppa-softmmu/memory.o
  CC      i386-softmmu/target/i386/excp_helper.o
  CC      i386-softmmu/target/i386/fpu_helper.o
  CC      lm32-softmmu/tcg/tcg.o
  CC      hppa-softmmu/memory_mapping.o
  CC      hppa-softmmu/dump.o
  CC      hppa-softmmu/migration/ram.o
  CC      lm32-softmmu/tcg/tcg-op.o
  CC      i386-softmmu/target/i386/int_helper.o
  CC      hppa-softmmu/accel/accel.o
  CC      i386-softmmu/target/i386/mem_helper.o
  CC      hppa-softmmu/accel/stubs/hax-stub.o
  CC      hppa-softmmu/accel/stubs/hvf-stub.o
  CC      i386-softmmu/target/i386/misc_helper.o
  CC      hppa-softmmu/accel/stubs/whpx-stub.o
  CC      hppa-softmmu/accel/stubs/kvm-stub.o
  CC      i386-softmmu/target/i386/mpx_helper.o
  CC      hppa-softmmu/accel/tcg/tcg-all.o
  CC      hppa-softmmu/accel/tcg/cputlb.o
  CC      i386-softmmu/target/i386/seg_helper.o
  CC      lm32-softmmu/tcg/tcg-op-vec.o
  CC      lm32-softmmu/tcg/tcg-op-gvec.o
  CC      i386-softmmu/target/i386/smm_helper.o
  CC      lm32-softmmu/tcg/tcg-common.o
  CC      i386-softmmu/target/i386/svm_helper.o
  CC      lm32-softmmu/tcg/optimize.o
  CC      hppa-softmmu/accel/tcg/tcg-runtime.o
  CC      i386-softmmu/target/i386/machine.o
  CC      hppa-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      lm32-softmmu/fpu/softfloat.o
  CC      i386-softmmu/target/i386/arch_memory_mapping.o
  CC      i386-softmmu/target/i386/arch_dump.o
  CC      i386-softmmu/target/i386/monitor.o
  CC      hppa-softmmu/accel/tcg/cpu-exec.o
  CC      i386-softmmu/target/i386/kvm-stub.o
  CC      i386-softmmu/target/i386/sev-stub.o
  GEN     trace/generated-helpers.c
  CC      i386-softmmu/trace/control-target.o
  CC      hppa-softmmu/accel/tcg/cpu-exec-common.o
  CC      hppa-softmmu/accel/tcg/translate-all.o
  CC      i386-softmmu/gdbstub-xml.o
  CC      i386-softmmu/trace/generated-helpers.o
  LINK    i386-softmmu/qemu-system-i386
  CC      hppa-softmmu/accel/tcg/translator.o
  CC      hppa-softmmu/hw/9pfs/virtio-9p-device.o
  CC      hppa-softmmu/hw/block/virtio-blk.o
  CC      hppa-softmmu/hw/block/vhost-user-blk.o
  CC      hppa-softmmu/hw/block/dataplane/virtio-blk.o
  CC      hppa-softmmu/hw/char/virtio-serial-bus.o
  CC      lm32-softmmu/disas.o
  CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
  CC      lm32-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      lm32-softmmu/cpus.o
  CC      hppa-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      hppa-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      hppa-softmmu/hw/display/vga.o
  GEN     m68k-softmmu/hmp-commands.h
  GEN     m68k-softmmu/hmp-commands-info.h
  GEN     m68k-softmmu/config-target.h
  CC      m68k-softmmu/exec.o
  CC      hppa-softmmu/hw/display/virtio-gpu.o
  CC      hppa-softmmu/hw/display/virtio-gpu-3d.o
  CC      m68k-softmmu/tcg/tcg.o
  CC      hppa-softmmu/hw/display/virtio-gpu-pci.o
  CC      hppa-softmmu/hw/display/virtio-vga.o
  CC      hppa-softmmu/hw/misc/ivshmem.o
  CC      hppa-softmmu/hw/misc/mmio_interface.o
  CC      hppa-softmmu/hw/net/virtio-net.o
  CC      m68k-softmmu/tcg/tcg-op.o
  CC      hppa-softmmu/hw/net/vhost_net.o
  CC      hppa-softmmu/hw/scsi/virtio-scsi.o
  CC      hppa-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      hppa-softmmu/hw/scsi/vhost-scsi-common.o
  CC      hppa-softmmu/hw/scsi/vhost-scsi.o
  CC      hppa-softmmu/hw/scsi/vhost-user-scsi.o
  CC      hppa-softmmu/hw/timer/mc146818rtc.o
  CC      m68k-softmmu/tcg/tcg-op-vec.o
  CC      hppa-softmmu/hw/vfio/common.o
  CC      m68k-softmmu/tcg/tcg-op-gvec.o
  CC      hppa-softmmu/hw/vfio/pci.o
  CC      m68k-softmmu/tcg/tcg-common.o
  CC      m68k-softmmu/tcg/optimize.o
  CC      hppa-softmmu/hw/vfio/pci-quirks.o
  CC      m68k-softmmu/fpu/softfloat.o
  CC      hppa-softmmu/hw/vfio/display.o
  CC      hppa-softmmu/hw/vfio/platform.o
  CC      hppa-softmmu/hw/vfio/spapr.o
  CC      hppa-softmmu/hw/virtio/virtio.o
  CC      hppa-softmmu/hw/virtio/virtio-balloon.o
  CC      hppa-softmmu/hw/virtio/virtio-crypto.o
  CC      hppa-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      hppa-softmmu/hw/virtio/vhost.o
  CC      m68k-softmmu/disas.o
  GEN     m68k-softmmu/gdbstub-xml.c
  CC      m68k-softmmu/arch_init.o
  CC      hppa-softmmu/hw/virtio/vhost-backend.o
  CC      m68k-softmmu/cpus.o
  CC      hppa-softmmu/hw/virtio/vhost-user.o
  CC      m68k-softmmu/monitor.o
  CC      hppa-softmmu/hw/virtio/vhost-vsock.o
  CC      hppa-softmmu/hw/hppa/machine.o
  CC      hppa-softmmu/hw/hppa/pci.o
  CC      hppa-softmmu/hw/hppa/dino.o
  CC      hppa-softmmu/target/hppa/translate.o
  CC      m68k-softmmu/gdbstub.o
  CC      m68k-softmmu/balloon.o
  CC      m68k-softmmu/ioport.o
  CC      m68k-softmmu/numa.o
  CC      m68k-softmmu/qtest.o
  CC      hppa-softmmu/target/hppa/helper.o
  CC      hppa-softmmu/target/hppa/cpu.o
  CC      m68k-softmmu/memory.o
  CC      hppa-softmmu/target/hppa/op_helper.o
  CC      hppa-softmmu/target/hppa/gdbstub.o
  CC      hppa-softmmu/target/hppa/mem_helper.o
  CC      hppa-softmmu/target/hppa/int_helper.o
  CC      m68k-softmmu/memory_mapping.o
  CC      hppa-softmmu/target/hppa/machine.o
  CC      m68k-softmmu/dump.o
  GEN     trace/generated-helpers.c
  CC      hppa-softmmu/trace/control-target.o
  CC      hppa-softmmu/trace/generated-helpers.o
  LINK    hppa-softmmu/qemu-system-hppa
  CC      m68k-softmmu/migration/ram.o
  CC      m68k-softmmu/accel/accel.o
  CC      m68k-softmmu/accel/stubs/hax-stub.o
  CC      m68k-softmmu/accel/stubs/hvf-stub.o
  CC      lm32-softmmu/monitor.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      m68k-softmmu/accel/stubs/whpx-stub.o
  GEN     microblazeel-softmmu/hmp-commands.h
  CC      aarch64-softmmu/hw/misc/imx7_ccm.o
  GEN     microblazeel-softmmu/hmp-commands-info.h
  GEN     microblazeel-softmmu/config-target.h
  CC      microblazeel-softmmu/exec.o
  CC      m68k-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/hw/misc/imx2_wdt.o
  CC      m68k-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/hw/misc/imx7_snvs.o
  CC      m68k-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/hw/misc/imx7_gpr.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      microblazeel-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
  CC      m68k-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      microblazeel-softmmu/tcg/tcg-op.o
  CC      m68k-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      aarch64-softmmu/hw/misc/mps2-fpgaio.o
  CC      aarch64-softmmu/hw/misc/mps2-scc.o
  CC      aarch64-softmmu/hw/misc/tz-mpc.o
  CC      m68k-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/hw/misc/tz-ppc.o
  CC      m68k-softmmu/accel/tcg/cpu-exec-common.o
  CC      m68k-softmmu/accel/tcg/translate-all.o
  CC      aarch64-softmmu/hw/misc/iotkit-secctl.o
  CC      microblazeel-softmmu/tcg/tcg-op-vec.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      m68k-softmmu/accel/tcg/translator.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      microblazeel-softmmu/tcg/tcg-op-gvec.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      m68k-softmmu/hw/char/mcf_uart.o
  CC      aarch64-softmmu/hw/misc/mmio_interface.o
  CC      m68k-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/misc/msf2-sysreg.o
  CC      m68k-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      m68k-softmmu/hw/misc/mmio_interface.o
  CC      m68k-softmmu/hw/net/mcf_fec.o
  CC      m68k-softmmu/hw/net/vhost_net.o
  CC      microblazeel-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      m68k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      m68k-softmmu/hw/vfio/common.o
  CC      microblazeel-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      m68k-softmmu/hw/vfio/platform.o
  CC      microblazeel-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      m68k-softmmu/hw/vfio/spapr.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      m68k-softmmu/hw/m68k/an5206.o
  CC      aarch64-softmmu/hw/sd/bcm2835_sdhost.o
  CC      m68k-softmmu/hw/m68k/mcf5208.o
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      m68k-softmmu/hw/m68k/mcf5206.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      m68k-softmmu/hw/m68k/mcf_intc.o
  CC      m68k-softmmu/target/m68k/m68k-semi.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      m68k-softmmu/target/m68k/translate.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      microblazeel-softmmu/disas.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      microblazeel-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      microblazeel-softmmu/cpus.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      aarch64-softmmu/hw/usb/chipidea.o
  CC      microblazeel-softmmu/monitor.o
  CC      aarch64-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/vfio/pci.o
  CC      m68k-softmmu/target/m68k/op_helper.o
  CC      microblazeel-softmmu/gdbstub.o
  CC      m68k-softmmu/target/m68k/helper.o
  CC      aarch64-softmmu/hw/vfio/pci-quirks.o
  CC      microblazeel-softmmu/balloon.o
  CC      m68k-softmmu/target/m68k/cpu.o
  CC      microblazeel-softmmu/ioport.o
  CC      m68k-softmmu/target/m68k/fpu_helper.o
  CC      microblazeel-softmmu/numa.o
  CC      aarch64-softmmu/hw/vfio/display.o
  CC      microblazeel-softmmu/qtest.o
  CC      m68k-softmmu/target/m68k/softfloat.o
  CC      aarch64-softmmu/hw/vfio/platform.o
  CC      microblazeel-softmmu/memory.o
  CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC      m68k-softmmu/target/m68k/gdbstub.o
  CC      m68k-softmmu/target/m68k/monitor.o
  CC      aarch64-softmmu/hw/vfio/spapr.o
  GEN     trace/generated-helpers.c
  CC      m68k-softmmu/trace/control-target.o
  CC      microblazeel-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      m68k-softmmu/gdbstub-xml.o
  CC      microblazeel-softmmu/dump.o
  CC      m68k-softmmu/trace/generated-helpers.o
  LINK    m68k-softmmu/qemu-system-m68k
  CC      microblazeel-softmmu/migration/ram.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
  GEN     microblaze-softmmu/hmp-commands.h
  GEN     microblaze-softmmu/hmp-commands-info.h
  GEN     microblaze-softmmu/config-target.h
  CC      microblaze-softmmu/exec.o
  CC      aarch64-softmmu/hw/virtio/vhost.o
  CC      aarch64-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC      microblaze-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/arm/boot.o
  CC      aarch64-softmmu/hw/arm/virt.o
  CC      aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC      aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC      microblaze-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/arm/digic_boards.o
  CC      aarch64-softmmu/hw/arm/exynos4_boards.o
  CC      aarch64-softmmu/hw/arm/highbank.o
  CC      aarch64-softmmu/hw/arm/integratorcp.o
  CC      aarch64-softmmu/hw/arm/mainstone.o
  CC      aarch64-softmmu/hw/arm/musicpal.o
  CC      microblaze-softmmu/tcg/tcg-op-vec.o
  CC      aarch64-softmmu/hw/arm/netduino2.o
  CC      microblaze-softmmu/tcg/tcg-op-gvec.o
  CC      aarch64-softmmu/hw/arm/nseries.o
  CC      aarch64-softmmu/hw/arm/omap_sx1.o
  CC      aarch64-softmmu/hw/arm/palm.o
  CC      aarch64-softmmu/hw/arm/gumstix.o
  CC      microblaze-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/hw/arm/spitz.o
  CC      aarch64-softmmu/hw/arm/tosa.o
  CC      microblaze-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/hw/arm/z2.o
  CC      aarch64-softmmu/hw/arm/realview.o
  CC      aarch64-softmmu/hw/arm/stellaris.o
  CC      aarch64-softmmu/hw/arm/collie.o
  CC      aarch64-softmmu/hw/arm/vexpress.o
  CC      aarch64-softmmu/hw/arm/versatilepb.o
  CC      aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC      aarch64-softmmu/hw/arm/armv7m.o
  CC      aarch64-softmmu/hw/arm/exynos4210.o
  CC      aarch64-softmmu/hw/arm/pxa2xx.o
  CC      microblaze-softmmu/disas.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC      microblaze-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC      microblaze-softmmu/cpus.o
  CC      aarch64-softmmu/hw/arm/digic.o
  CC      aarch64-softmmu/hw/arm/omap1.o
  CC      microblaze-softmmu/monitor.o
  CC      aarch64-softmmu/hw/arm/omap2.o
  CC      microblaze-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/arm/strongarm.o
  CC      aarch64-softmmu/hw/arm/allwinner-a10.o
  CC      microblaze-softmmu/balloon.o
  CC      aarch64-softmmu/hw/arm/cubieboard.o
  CC      microblaze-softmmu/ioport.o
  CC      aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC      microblaze-softmmu/numa.o
  CC      aarch64-softmmu/hw/arm/bcm2836.o
  CC      microblaze-softmmu/qtest.o
  CC      aarch64-softmmu/hw/arm/raspi.o
  CC      microblaze-softmmu/memory.o
  CC      aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC      aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC      aarch64-softmmu/hw/arm/xlnx-zcu102.o
  CC      microblaze-softmmu/memory_mapping.o
  CC      microblaze-softmmu/dump.o
  CC      aarch64-softmmu/hw/arm/fsl-imx25.o
  CC      aarch64-softmmu/hw/arm/imx25_pdk.o
  CC      aarch64-softmmu/hw/arm/fsl-imx31.o
  CC      microblaze-softmmu/migration/ram.o
  CC      aarch64-softmmu/hw/arm/kzm.o
  CC      aarch64-softmmu/hw/arm/fsl-imx6.o
  CC      aarch64-softmmu/hw/arm/sabrelite.o
  CC      aarch64-softmmu/hw/arm/aspeed_soc.o
  CC      microblaze-softmmu/accel/accel.o
  CC      aarch64-softmmu/hw/arm/aspeed.o
  CC      microblaze-softmmu/accel/stubs/hax-stub.o
  CC      microblaze-softmmu/accel/stubs/hvf-stub.o
  CC      aarch64-softmmu/hw/arm/mps2.o
  CC      microblaze-softmmu/accel/stubs/whpx-stub.o
  CC      microblaze-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/hw/arm/mps2-tz.o
  CC      microblaze-softmmu/accel/tcg/tcg-all.o
  CC      microblaze-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/hw/arm/msf2-soc.o
  CC      aarch64-softmmu/hw/arm/msf2-som.o
  CC      aarch64-softmmu/hw/arm/iotkit.o
  CC      aarch64-softmmu/hw/arm/fsl-imx7.o
  CC      aarch64-softmmu/hw/arm/mcimx7d-sabre.o
  CC      aarch64-softmmu/hw/arm/smmu-common.o
  CC      aarch64-softmmu/hw/arm/smmuv3.o
  CC      aarch64-softmmu/target/arm/arm-semi.o
  CC      microblaze-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64-softmmu/target/arm/machine.o
  CC      microblaze-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      aarch64-softmmu/target/arm/psci.o
  CC      aarch64-softmmu/target/arm/arch_dump.o
  CC      aarch64-softmmu/target/arm/monitor.o
  CC      aarch64-softmmu/target/arm/kvm-stub.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/target/arm/translate.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblaze-softmmu/accel/tcg/translate-all.o
  CC      microblaze-softmmu/accel/tcg/translator.o
  CC      microblaze-softmmu/hw/core/generic-loader.o
  CC      microblaze-softmmu/hw/core/null-machine.o
  CC      microblaze-softmmu/hw/dma/xlnx_dpdma.o
  CC      microblaze-softmmu/hw/misc/mmio_interface.o
  CC      microblaze-softmmu/hw/net/xilinx_ethlite.o
  CC      microblaze-softmmu/hw/net/vhost_net.o
  CC      microblaze-softmmu/hw/net/rocker/qmp-norocker.o
  CC      microblaze-softmmu/hw/vfio/common.o
  CC      microblaze-softmmu/hw/vfio/platform.o
  CC      microblaze-softmmu/hw/vfio/spapr.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      microblaze-softmmu/hw/microblaze/xlnx-zynqmp-pmu.o
  CC      microblaze-softmmu/hw/microblaze/boot.o
  CC      microblaze-softmmu/target/microblaze/translate.o
  CC      microblaze-softmmu/target/microblaze/op_helper.o
  CC      microblaze-softmmu/target/microblaze/helper.o
  CC      microblaze-softmmu/target/microblaze/cpu.o
  CC      aarch64-softmmu/target/arm/op_helper.o
  CC      microblaze-softmmu/target/microblaze/gdbstub.o
  CC      microblaze-softmmu/target/microblaze/mmu.o
  GEN     trace/generated-helpers.c
  CC      microblaze-softmmu/trace/control-target.o
  CC      microblaze-softmmu/trace/generated-helpers.o
  LINK    microblaze-softmmu/qemu-system-microblaze
  CC      aarch64-softmmu/target/arm/helper.o
  CC      lm32-softmmu/gdbstub.o
  CC      microblazeel-softmmu/accel/accel.o
  CC      lm32-softmmu/balloon.o
  CC      lm32-softmmu/ioport.o
  CC      microblazeel-softmmu/accel/stubs/hax-stub.o
  CC      lm32-softmmu/numa.o
  CC      lm32-softmmu/qtest.o
  GEN     mips64el-softmmu/hmp-commands.h
  GEN     mips64el-softmmu/hmp-commands-info.h
  GEN     mips64el-softmmu/config-target.h
  CC      mips64el-softmmu/exec.o
  CC      lm32-softmmu/memory.o
  CC      aarch64-softmmu/target/arm/cpu.o
  CC      lm32-softmmu/memory_mapping.o
  CC      lm32-softmmu/dump.o
  CC      aarch64-softmmu/target/arm/neon_helper.o
  CC      mips64el-softmmu/tcg/tcg.o
  CC      lm32-softmmu/migration/ram.o
  CC      aarch64-softmmu/target/arm/iwmmxt_helper.o
  CC      lm32-softmmu/accel/accel.o
  CC      lm32-softmmu/accel/stubs/hax-stub.o
  CC      lm32-softmmu/accel/stubs/hvf-stub.o
  CC      aarch64-softmmu/target/arm/vec_helper.o
  CC      lm32-softmmu/accel/stubs/whpx-stub.o
  CC      lm32-softmmu/accel/stubs/kvm-stub.o
  CC      mips64el-softmmu/tcg/tcg-op.o
  CC      lm32-softmmu/accel/tcg/tcg-all.o
  CC      lm32-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/target/arm/gdbstub.o
  CC      aarch64-softmmu/target/arm/cpu64.o
  CC      aarch64-softmmu/target/arm/translate-a64.o
  CC      mips64el-softmmu/tcg/tcg-op-vec.o
  CC      mips64el-softmmu/tcg/tcg-op-gvec.o
  CC      lm32-softmmu/accel/tcg/tcg-runtime.o
  CC      lm32-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      lm32-softmmu/accel/tcg/cpu-exec.o
  CC      lm32-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64el-softmmu/tcg/tcg-common.o
  CC      lm32-softmmu/accel/tcg/translate-all.o
  CC      mips64el-softmmu/tcg/optimize.o
  CC      lm32-softmmu/accel/tcg/translator.o
  CC      lm32-softmmu/hw/core/generic-loader.o
  CC      lm32-softmmu/hw/core/null-machine.o
  CC      mips64el-softmmu/fpu/softfloat.o
  CC      lm32-softmmu/hw/input/milkymist-softusb.o
  CC      lm32-softmmu/hw/misc/milkymist-hpdmc.o
  CC      lm32-softmmu/hw/misc/milkymist-pfpu.o
  CC      aarch64-softmmu/target/arm/helper-a64.o
  CC      lm32-softmmu/hw/misc/mmio_interface.o
  CC      lm32-softmmu/hw/net/milkymist-minimac2.o
  CC      aarch64-softmmu/target/arm/gdbstub64.o
  CC      aarch64-softmmu/target/arm/crypto_helper.o
  CC      lm32-softmmu/hw/net/vhost_net.o
  CC      lm32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      lm32-softmmu/hw/sd/milkymist-memcard.o
  CC      aarch64-softmmu/target/arm/arm-powerctl.o
  CC      lm32-softmmu/hw/vfio/common.o
  GEN     aarch64-softmmu/target/arm/decode-sve.inc.c
  CC      aarch64-softmmu/target/arm/sve_helper.o
  CC      lm32-softmmu/hw/vfio/platform.o
  CC      lm32-softmmu/hw/vfio/spapr.o
  CC      lm32-softmmu/hw/lm32/lm32_boards.o
  CC      mips64el-softmmu/disas.o
  CC      lm32-softmmu/hw/lm32/milkymist.o
  CC      mips64el-softmmu/arch_init.o
  CC      mips64el-softmmu/cpus.o
  CC      lm32-softmmu/target/lm32/translate.o
  CC      mips64el-softmmu/monitor.o
  CC      lm32-softmmu/target/lm32/op_helper.o
  CC      lm32-softmmu/target/lm32/helper.o
  CC      lm32-softmmu/target/lm32/cpu.o
  CC      lm32-softmmu/target/lm32/gdbstub.o
  CC      lm32-softmmu/target/lm32/lm32-semi.o
  CC      lm32-softmmu/target/lm32/machine.o
  CC      mips64el-softmmu/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      lm32-softmmu/trace/control-target.o
  CC      lm32-softmmu/trace/generated-helpers.o
  LINK    lm32-softmmu/qemu-system-lm32
  CC      mips64el-softmmu/balloon.o
  CC      mips64el-softmmu/ioport.o
  CC      mips64el-softmmu/numa.o
  CC      mips64el-softmmu/qtest.o
  CC      microblazeel-softmmu/accel/stubs/hvf-stub.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/trace/control-target.o
  CC      mips64el-softmmu/memory.o
  CC      microblazeel-softmmu/accel/stubs/whpx-stub.o
  CC      aarch64-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/target/arm/translate-sve.o
  GEN     mips64-softmmu/hmp-commands.h
  CC      microblazeel-softmmu/accel/stubs/kvm-stub.o
  GEN     mips64-softmmu/hmp-commands-info.h
  GEN     mips64-softmmu/config-target.h
  CC      microblazeel-softmmu/accel/tcg/tcg-all.o
  CC      mips64-softmmu/exec.o
  CC      microblazeel-softmmu/accel/tcg/cputlb.o
  CC      mips64el-softmmu/memory_mapping.o
  CC      mips64el-softmmu/dump.o
  CC      mips64-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/migration/ram.o
  CC      microblazeel-softmmu/accel/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      mips64el-softmmu/accel/accel.o
  CC      mips64el-softmmu/accel/stubs/hax-stub.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec.o
  CC      mips64el-softmmu/accel/stubs/hvf-stub.o
  CC      mips64-softmmu/tcg/tcg-op.o
  CC      mips64el-softmmu/accel/stubs/whpx-stub.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblazeel-softmmu/accel/tcg/translate-all.o
  CC      mips64el-softmmu/accel/stubs/kvm-stub.o
  CC      mips64el-softmmu/accel/tcg/tcg-all.o
  CC      microblazeel-softmmu/accel/tcg/translator.o
  CC      mips64el-softmmu/accel/tcg/cputlb.o
  CC      microblazeel-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/trace/generated-helpers.o
  CC      mips64-softmmu/tcg/tcg-op-vec.o
  CC      microblazeel-softmmu/hw/core/null-machine.o
  CC      microblazeel-softmmu/hw/dma/xlnx_dpdma.o
  LINK    aarch64-softmmu/qemu-system-aarch64
  CC      mips64-softmmu/tcg/tcg-op-gvec.o
  CC      microblazeel-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-softmmu/hw/net/xilinx_ethlite.o
  CC      microblazeel-softmmu/hw/net/vhost_net.o
  CC      mips64el-softmmu/accel/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/hw/net/rocker/qmp-norocker.o
  CC      mips64el-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      microblazeel-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/tcg/tcg-common.o
  CC      mips64-softmmu/tcg/optimize.o
  GEN     mipsel-softmmu/hmp-commands.h
  GEN     mipsel-softmmu/hmp-commands-info.h
  GEN     mipsel-softmmu/config-target.h
  CC      mipsel-softmmu/exec.o
  CC      mips64-softmmu/fpu/softfloat.o
  CC      mipsel-softmmu/tcg/tcg.o
  CC      mipsel-softmmu/tcg/tcg-op.o
  CC      mips64-softmmu/disas.o
  CC      mips64-softmmu/arch_init.o
  CC      mips64-softmmu/cpus.o
  CC      mips64-softmmu/monitor.o
  CC      mipsel-softmmu/tcg/tcg-op-vec.o
  CC      mips64-softmmu/gdbstub.o
  CC      mipsel-softmmu/tcg/tcg-op-gvec.o
  CC      mips64-softmmu/balloon.o
  CC      mips64-softmmu/ioport.o
  CC      mips64-softmmu/numa.o
  CC      mips64-softmmu/qtest.o
  CC      mipsel-softmmu/tcg/tcg-common.o
  CC      mips64-softmmu/memory.o
  CC      mipsel-softmmu/tcg/optimize.o
  CC      mipsel-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/memory_mapping.o
  CC      mips64-softmmu/dump.o
  CC      mips64-softmmu/migration/ram.o
  CC      mips64-softmmu/accel/accel.o
  CC      mips64-softmmu/accel/stubs/hax-stub.o
  CC      mips64-softmmu/accel/stubs/hvf-stub.o
  CC      mips64-softmmu/accel/stubs/whpx-stub.o
  CC      mips64-softmmu/accel/stubs/kvm-stub.o
  CC      mips64-softmmu/accel/tcg/tcg-all.o
  CC      mips64-softmmu/accel/tcg/cputlb.o
  CC      mipsel-softmmu/disas.o
  CC      mipsel-softmmu/arch_init.o
  CC      mipsel-softmmu/cpus.o
  CC      mipsel-softmmu/monitor.o
  CC      mips64-softmmu/accel/tcg/tcg-runtime.o
  CC      mips64-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      mipsel-softmmu/gdbstub.o
  CC      mipsel-softmmu/balloon.o
  CC      mipsel-softmmu/ioport.o
  CC      mips64-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/numa.o
  CC      mips64-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/qtest.o
  CC      mips64-softmmu/accel/tcg/translate-all.o
  CC      mipsel-softmmu/memory.o
  CC      mips64-softmmu/accel/tcg/translator.o
  CC      mips64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mips64-softmmu/hw/block/virtio-blk.o
  CC      mipsel-softmmu/memory_mapping.o
  CC      mipsel-softmmu/dump.o
  CC      mips64-softmmu/hw/block/vhost-user-blk.o
  CC      mips64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsel-softmmu/migration/ram.o
  CC      mips64-softmmu/hw/char/virtio-serial-bus.o
  CC      mips64-softmmu/hw/core/generic-loader.o
  CC      mips64-softmmu/hw/core/null-machine.o
  CC      mips64-softmmu/hw/display/vga.o
  CC      mipsel-softmmu/accel/accel.o
  CC      mipsel-softmmu/accel/stubs/hax-stub.o
  CC      mipsel-softmmu/accel/stubs/hvf-stub.o
  CC      mipsel-softmmu/accel/stubs/whpx-stub.o
  CC      mipsel-softmmu/accel/stubs/kvm-stub.o
  CC      mips64-softmmu/hw/display/virtio-gpu.o
  CC      mipsel-softmmu/accel/tcg/tcg-all.o
  CC      mipsel-softmmu/accel/tcg/cputlb.o
  CC      mips64-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64-softmmu/hw/intc/mips_gic.o
  CC      mips64-softmmu/hw/misc/ivshmem.o
  CC      mips64-softmmu/hw/misc/mips_cmgcr.o
  CC      mips64-softmmu/hw/misc/mips_cpc.o
  CC      mips64-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/hw/misc/mmio_interface.o
  CC      mipsel-softmmu/accel/tcg/tcg-runtime.o
  CC      mips64-softmmu/hw/net/virtio-net.o
  CC      mipsel-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      mips64-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mipsel-softmmu/accel/tcg/translate-all.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mipsel-softmmu/accel/tcg/translator.o
  CC      mips64-softmmu/hw/timer/mips_gictimer.o
  CC      mips64-softmmu/hw/timer/mc146818rtc.o
  CC      mipsel-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mips64-softmmu/hw/vfio/common.o
  CC      mipsel-softmmu/hw/block/virtio-blk.o
  CC      mipsel-softmmu/hw/block/vhost-user-blk.o
  CC      mips64-softmmu/hw/vfio/pci.o
  CC      mipsel-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsel-softmmu/hw/char/virtio-serial-bus.o
  CC      mipsel-softmmu/hw/core/generic-loader.o
  CC      mipsel-softmmu/hw/core/null-machine.o
  CC      mips64-softmmu/hw/vfio/pci-quirks.o
  CC      mipsel-softmmu/hw/display/vga.o
  CC      mips64-softmmu/hw/vfio/display.o
  CC      mips64-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/display/virtio-gpu.o
  CC      mips64-softmmu/hw/vfio/spapr.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64-softmmu/hw/virtio/virtio.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-pci.o
  CC      mipsel-softmmu/hw/intc/mips_gic.o
  CC      mips64-softmmu/hw/virtio/virtio-balloon.o
  CC      mipsel-softmmu/hw/misc/ivshmem.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-softmmu/hw/misc/mips_cpc.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mipsel-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/misc/mmio_interface.o
  CC      mipsel-softmmu/hw/net/virtio-net.o
  CC      mips64-softmmu/hw/virtio/vhost-backend.o
  CC      mips64-softmmu/hw/virtio/vhost-user.o
  CC      mipsel-softmmu/hw/net/vhost_net.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi.o
  CC      mips64-softmmu/hw/virtio/vhost-vsock.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64-softmmu/hw/mips/mips_r4k.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi.o
  CC      mipsel-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips64-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/timer/mips_gictimer.o
  CC      mips64-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/hw/timer/mc146818rtc.o
  CC      mips64-softmmu/hw/mips/mips_int.o
  CC      mips64-softmmu/hw/mips/mips_jazz.o
  CC      mipsel-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/hw/mips/gt64xxx_pci.o
  CC      mips64-softmmu/hw/mips/cps.o
  CC      mips64-softmmu/target/mips/translate.o
  CC      mipsel-softmmu/hw/vfio/pci.o
  CC      mipsel-softmmu/hw/vfio/pci-quirks.o
  CC      mipsel-softmmu/hw/vfio/display.o
  CC      mipsel-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/vfio/spapr.o
  CC      mipsel-softmmu/hw/virtio/virtio.o
  CC      mipsel-softmmu/hw/virtio/virtio-balloon.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mipsel-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hw/virtio/vhost-user.o
  CC      mipsel-softmmu/hw/virtio/vhost-vsock.o
  CC      mipsel-softmmu/hw/mips/mips_r4k.o
  CC      mipsel-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/hw/mips/mips_int.o
  CC      mipsel-softmmu/hw/mips/gt64xxx_pci.o
  CC      mipsel-softmmu/hw/mips/cps.o
  CC      mipsel-softmmu/target/mips/translate.o
  CC      mips64-softmmu/target/mips/dsp_helper.o
  CC      mips64-softmmu/target/mips/op_helper.o
  CC      mips64-softmmu/target/mips/lmi_helper.o
  CC      mips64-softmmu/target/mips/helper.o
  CC      mips64-softmmu/target/mips/cpu.o
  CC      mips64-softmmu/target/mips/gdbstub.o
  CC      mips64-softmmu/target/mips/msa_helper.o
  CC      mipsel-softmmu/target/mips/dsp_helper.o
  CC      mipsel-softmmu/target/mips/op_helper.o
  CC      mips64-softmmu/target/mips/mips-semi.o
  CC      mips64-softmmu/target/mips/machine.o
  CC      mips64-softmmu/target/mips/cp0_timer.o
  GEN     trace/generated-helpers.c
  CC      mips64-softmmu/trace/control-target.o
  CC      mips64-softmmu/trace/generated-helpers.o
  CC      mipsel-softmmu/target/mips/lmi_helper.o
  LINK    mips64-softmmu/qemu-system-mips64
  CC      mipsel-softmmu/target/mips/helper.o
  CC      mipsel-softmmu/target/mips/cpu.o
  CC      mipsel-softmmu/target/mips/gdbstub.o
  CC      mipsel-softmmu/target/mips/msa_helper.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/target/mips/mips-semi.o
  CC      mipsel-softmmu/target/mips/machine.o
  CC      mipsel-softmmu/target/mips/cp0_timer.o
  GEN     trace/generated-helpers.c
  CC      mipsel-softmmu/trace/control-target.o
  CC      microblazeel-softmmu/hw/vfio/platform.o
  CC      microblazeel-softmmu/hw/vfio/spapr.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      mips64el-softmmu/accel/tcg/translate-all.o
  GEN     mips-softmmu/hmp-commands.h
  CC      microblazeel-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  GEN     mips-softmmu/hmp-commands-info.h
  GEN     mips-softmmu/config-target.h
  CC      microblazeel-softmmu/hw/microblaze/xlnx-zynqmp-pmu.o
  CC      mips-softmmu/exec.o
  CC      mips64el-softmmu/accel/tcg/translator.o
  CC      microblazeel-softmmu/hw/microblaze/boot.o
  CC      mipsel-softmmu/trace/generated-helpers.o
  CC      microblazeel-softmmu/target/microblaze/translate.o
  CC      mips64el-softmmu/hw/9pfs/virtio-9p-device.o
  LINK    mipsel-softmmu/qemu-system-mipsel
  CC      mips64el-softmmu/hw/block/virtio-blk.o
  CC      mips-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/block/vhost-user-blk.o
  CC      microblazeel-softmmu/target/microblaze/op_helper.o
  CC      mips64el-softmmu/hw/block/dataplane/virtio-blk.o
  CC      microblazeel-softmmu/target/microblaze/helper.o
  CC      microblazeel-softmmu/target/microblaze/cpu.o
  CC      mips64el-softmmu/hw/char/virtio-serial-bus.o
  CC      mips64el-softmmu/hw/core/generic-loader.o
  GEN     moxie-softmmu/hmp-commands.h
  GEN     moxie-softmmu/hmp-commands-info.h
  GEN     moxie-softmmu/config-target.h
  CC      moxie-softmmu/exec.o
  CC      mips64el-softmmu/hw/core/null-machine.o
  CC      mips64el-softmmu/hw/display/vga.o
  CC      mips-softmmu/tcg/tcg-op.o
  CC      mips64el-softmmu/hw/display/virtio-gpu.o
  CC      moxie-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips-softmmu/tcg/tcg-op-vec.o
  CC      mips64el-softmmu/hw/intc/mips_gic.o
  CC      mips64el-softmmu/hw/misc/ivshmem.o
  CC      mips-softmmu/tcg/tcg-op-gvec.o
  CC      mips64el-softmmu/hw/misc/mips_cmgcr.o
  CC      moxie-softmmu/tcg/tcg-op.o
  CC      mips64el-softmmu/hw/misc/mips_cpc.o
  CC      mips64el-softmmu/hw/misc/mips_itu.o
  CC      mips64el-softmmu/hw/misc/mmio_interface.o
  CC      mips64el-softmmu/hw/net/virtio-net.o
  CC      mips-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi.o
  CC      moxie-softmmu/tcg/tcg-op-vec.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      moxie-softmmu/tcg/tcg-op-gvec.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips-softmmu/fpu/softfloat.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64el-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips64el-softmmu/hw/timer/mips_gictimer.o
  CC      mips64el-softmmu/hw/timer/mc146818rtc.o
  CC      moxie-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/vfio/common.o
  CC      moxie-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/hw/vfio/pci.o
  CC      moxie-softmmu/fpu/softfloat.o
  CC      mips64el-softmmu/hw/vfio/pci-quirks.o
  CC      mips-softmmu/disas.o
  CC      mips64el-softmmu/hw/vfio/display.o
  CC      mips-softmmu/arch_init.o
  CC      mips64el-softmmu/hw/vfio/platform.o
  CC      mips-softmmu/cpus.o
  CC      mips64el-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/monitor.o
  CC      mips64el-softmmu/hw/virtio/virtio.o
  CC      moxie-softmmu/disas.o
  CC      mips64el-softmmu/hw/virtio/virtio-balloon.o
  CC      moxie-softmmu/arch_init.o
  CC      mips-softmmu/gdbstub.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto.o
  CC      moxie-softmmu/cpus.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      moxie-softmmu/monitor.o
  CC      mips-softmmu/balloon.o
  CC      mips64el-softmmu/hw/virtio/vhost.o
  CC      mips-softmmu/ioport.o
  CC      mips-softmmu/numa.o
  CC      mips64el-softmmu/hw/virtio/vhost-backend.o
  CC      mips-softmmu/qtest.o
  CC      mips64el-softmmu/hw/virtio/vhost-user.o
  CC      mips-softmmu/memory.o
  CC      moxie-softmmu/gdbstub.o
  CC      mips64el-softmmu/hw/virtio/vhost-vsock.o
  CC      mips64el-softmmu/hw/mips/mips_r4k.o
  CC      mips64el-softmmu/hw/mips/mips_malta.o
  CC      moxie-softmmu/balloon.o
  CC      moxie-softmmu/ioport.o
  CC      mips-softmmu/memory_mapping.o
  CC      mips64el-softmmu/hw/mips/mips_mipssim.o
  CC      moxie-softmmu/numa.o
  CC      mips-softmmu/dump.o
  CC      mips64el-softmmu/hw/mips/addr.o
  CC      mips64el-softmmu/hw/mips/mips_int.o
  CC      moxie-softmmu/qtest.o
  CC      mips64el-softmmu/hw/mips/mips_jazz.o
  CC      mips-softmmu/migration/ram.o
  CC      moxie-softmmu/memory.o
  CC      mips64el-softmmu/hw/mips/mips_fulong2e.o
  CC      mips64el-softmmu/hw/mips/gt64xxx_pci.o
  CC      mips64el-softmmu/hw/mips/cps.o
  CC      mips64el-softmmu/hw/mips/boston.o
  CC      mips-softmmu/accel/accel.o
  CC      moxie-softmmu/memory_mapping.o
  CC      mips64el-softmmu/target/mips/translate.o
  CC      mips-softmmu/accel/stubs/hax-stub.o
  CC      moxie-softmmu/dump.o
  CC      mips-softmmu/accel/stubs/hvf-stub.o
  CC      mips-softmmu/accel/stubs/whpx-stub.o
  CC      mips-softmmu/accel/stubs/kvm-stub.o
  CC      moxie-softmmu/migration/ram.o
  CC      mips-softmmu/accel/tcg/tcg-all.o
  CC      mips-softmmu/accel/tcg/cputlb.o
  CC      moxie-softmmu/accel/accel.o
  CC      moxie-softmmu/accel/stubs/hax-stub.o
  CC      moxie-softmmu/accel/stubs/hvf-stub.o
  CC      moxie-softmmu/accel/stubs/whpx-stub.o
  CC      moxie-softmmu/accel/stubs/kvm-stub.o
  CC      moxie-softmmu/accel/tcg/tcg-all.o
  CC      moxie-softmmu/accel/tcg/cputlb.o
  CC      mips-softmmu/accel/tcg/tcg-runtime.o
  CC      mips-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      mips-softmmu/accel/tcg/cpu-exec.o
  CC      mips-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips-softmmu/accel/tcg/translate-all.o
  CC      moxie-softmmu/accel/tcg/tcg-runtime.o
  CC      moxie-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      mips-softmmu/accel/tcg/translator.o
  CC      mips-softmmu/hw/9pfs/virtio-9p-device.o
  CC      moxie-softmmu/accel/tcg/cpu-exec.o
  CC      mips-softmmu/hw/block/virtio-blk.o
  CC      moxie-softmmu/accel/tcg/cpu-exec-common.o
  CC      moxie-softmmu/accel/tcg/translate-all.o
  CC      mips-softmmu/hw/block/vhost-user-blk.o
  CC      mips-softmmu/hw/block/dataplane/virtio-blk.o
  CC      moxie-softmmu/accel/tcg/translator.o
  CC      mips-softmmu/hw/char/virtio-serial-bus.o
  CC      moxie-softmmu/hw/core/generic-loader.o
  CC      moxie-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/core/generic-loader.o
  CC      moxie-softmmu/hw/display/vga.o
  CC      mips-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/display/vga.o
  CC      moxie-softmmu/hw/misc/mmio_interface.o
  CC      moxie-softmmu/hw/net/vhost_net.o
  CC      mips64el-softmmu/target/mips/dsp_helper.o
  CC      mips-softmmu/hw/display/virtio-gpu.o
  CC      moxie-softmmu/hw/net/rocker/qmp-norocker.o
  CC      moxie-softmmu/hw/timer/mc146818rtc.o
  CC      moxie-softmmu/hw/vfio/common.o
  CC      mips-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips-softmmu/hw/display/virtio-gpu-pci.o
  CC      moxie-softmmu/hw/vfio/platform.o
  CC      mips64el-softmmu/target/mips/op_helper.o
  CC      mips-softmmu/hw/intc/mips_gic.o
  CC      moxie-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hw/misc/ivshmem.o
  CC      moxie-softmmu/hw/moxie/moxiesim.o
  CC      mips-softmmu/hw/misc/mips_cmgcr.o
  CC      moxie-softmmu/target/moxie/translate.o
  CC      mips-softmmu/hw/misc/mips_cpc.o
  CC      mips-softmmu/hw/misc/mips_itu.o
  CC      mips-softmmu/hw/misc/mmio_interface.o
  CC      moxie-softmmu/target/moxie/helper.o
  CC      mips-softmmu/hw/net/virtio-net.o
  CC      moxie-softmmu/target/moxie/machine.o
  CC      moxie-softmmu/target/moxie/cpu.o
  CC      moxie-softmmu/target/moxie/mmu.o
  CC      mips64el-softmmu/target/mips/lmi_helper.o
  GEN     trace/generated-helpers.c
  CC      moxie-softmmu/trace/control-target.o
  CC      mips-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/hw/scsi/virtio-scsi.o
  CC      moxie-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/target/mips/helper.o
  LINK    moxie-softmmu/qemu-system-moxie
  CC      mips-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64el-softmmu/target/mips/cpu.o
  CC      mips-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64el-softmmu/target/mips/gdbstub.o
  CC      mips-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64el-softmmu/target/mips/msa_helper.o
  CC      mips-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips-softmmu/hw/timer/mips_gictimer.o
  CC      microblazeel-softmmu/target/microblaze/gdbstub.o
  CC      mips-softmmu/hw/timer/mc146818rtc.o
  CC      microblazeel-softmmu/target/microblaze/mmu.o
  CC      mips-softmmu/hw/vfio/common.o
  GEN     trace/generated-helpers.c
  CC      microblazeel-softmmu/trace/control-target.o
  GEN     nios2-softmmu/hmp-commands.h
  GEN     nios2-softmmu/hmp-commands-info.h
  GEN     nios2-softmmu/config-target.h
  CC      microblazeel-softmmu/trace/generated-helpers.o
  CC      nios2-softmmu/exec.o
  LINK    microblazeel-softmmu/qemu-system-microblazeel
  CC      mips-softmmu/hw/vfio/pci.o
  CC      mips-softmmu/hw/vfio/pci-quirks.o
  GEN     or1k-softmmu/hmp-commands.h
  CC      mips64el-softmmu/target/mips/mips-semi.o
  GEN     or1k-softmmu/hmp-commands-info.h
  GEN     or1k-softmmu/config-target.h
  CC      or1k-softmmu/exec.o
  CC      mips64el-softmmu/target/mips/machine.o
  CC      mips-softmmu/hw/vfio/display.o
  CC      mips64el-softmmu/target/mips/cp0_timer.o
  GEN     trace/generated-helpers.c
  CC      mips64el-softmmu/trace/control-target.o
  CC      mips-softmmu/hw/vfio/platform.o
  CC      mips64el-softmmu/trace/generated-helpers.o
  CC      mips-softmmu/hw/vfio/spapr.o
  LINK    mips64el-softmmu/qemu-system-mips64el
  CC      mips-softmmu/hw/virtio/virtio.o
  CC      or1k-softmmu/tcg/tcg.o
  CC      mips-softmmu/hw/virtio/virtio-balloon.o
  GEN     ppc64-softmmu/hmp-commands.h
  GEN     ppc64-softmmu/hmp-commands-info.h
  GEN     ppc64-softmmu/config-target.h
  CC      ppc64-softmmu/exec.o
  CC      or1k-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/tcg/tcg.o
  CC      or1k-softmmu/tcg/tcg-op-vec.o
  CC      or1k-softmmu/tcg/tcg-op-gvec.o
  CC      ppc64-softmmu/tcg/tcg-op.o
  CC      or1k-softmmu/tcg/tcg-common.o
  CC      or1k-softmmu/tcg/optimize.o
  CC      or1k-softmmu/fpu/softfloat.o
  CC      ppc64-softmmu/tcg/tcg-op-vec.o
  CC      ppc64-softmmu/tcg/tcg-op-gvec.o
  CC      or1k-softmmu/disas.o
  CC      ppc64-softmmu/tcg/tcg-common.o
  CC      ppc64-softmmu/tcg/optimize.o
  CC      or1k-softmmu/arch_init.o
  CC      or1k-softmmu/cpus.o
  CC      or1k-softmmu/monitor.o
  CC      ppc64-softmmu/fpu/softfloat.o
  CC      or1k-softmmu/gdbstub.o
  CC      or1k-softmmu/balloon.o
  CC      or1k-softmmu/ioport.o
  CC      or1k-softmmu/numa.o
  CC      or1k-softmmu/qtest.o
  CC      or1k-softmmu/memory.o
  CC      ppc64-softmmu/disas.o
  GEN     ppc64-softmmu/gdbstub-xml.c
  CC      ppc64-softmmu/arch_init.o
  CC      ppc64-softmmu/cpus.o
  CC      or1k-softmmu/memory_mapping.o
  CC      ppc64-softmmu/monitor.o
  CC      or1k-softmmu/dump.o
  CC      or1k-softmmu/migration/ram.o
  CC      ppc64-softmmu/gdbstub.o
  CC      or1k-softmmu/accel/accel.o
  CC      or1k-softmmu/accel/stubs/hax-stub.o
  CC      ppc64-softmmu/balloon.o
  CC      or1k-softmmu/accel/stubs/hvf-stub.o
  CC      ppc64-softmmu/ioport.o
  CC      or1k-softmmu/accel/stubs/whpx-stub.o
  CC      or1k-softmmu/accel/stubs/kvm-stub.o
  CC      ppc64-softmmu/numa.o
  CC      or1k-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/accel/tcg/cputlb.o
  CC      ppc64-softmmu/qtest.o
  CC      ppc64-softmmu/memory.o
  CC      ppc64-softmmu/memory_mapping.o
  CC      ppc64-softmmu/dump.o
  CC      ppc64-softmmu/migration/ram.o
  CC      or1k-softmmu/accel/tcg/tcg-runtime.o
  CC      or1k-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      ppc64-softmmu/accel/accel.o
  CC      ppc64-softmmu/accel/stubs/hax-stub.o
  CC      or1k-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/accel/stubs/hvf-stub.o
  CC      ppc64-softmmu/accel/stubs/whpx-stub.o
  CC      or1k-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/accel/tcg/translate-all.o
  CC      ppc64-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-softmmu/accel/tcg/cputlb.o
  CC      or1k-softmmu/accel/tcg/translator.o
  CC      or1k-softmmu/hw/core/generic-loader.o
  CC      or1k-softmmu/hw/core/null-machine.o
  CC      or1k-softmmu/hw/intc/ompic.o
  CC      or1k-softmmu/hw/misc/mmio_interface.o
  CC      or1k-softmmu/hw/net/vhost_net.o
  CC      or1k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      or1k-softmmu/hw/vfio/common.o
  CC      or1k-softmmu/hw/vfio/platform.o
  CC      ppc64-softmmu/accel/tcg/tcg-runtime.o
  CC      or1k-softmmu/hw/vfio/spapr.o
  CC      ppc64-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      or1k-softmmu/hw/openrisc/pic_cpu.o
  CC      or1k-softmmu/hw/openrisc/cputimer.o
  CC      or1k-softmmu/hw/openrisc/openrisc_sim.o
  CC      or1k-softmmu/target/openrisc/machine.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec.o
  CC      or1k-softmmu/target/openrisc/cpu.o
  CC      or1k-softmmu/target/openrisc/exception.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec-common.o
  CC      or1k-softmmu/target/openrisc/interrupt.o
  CC      ppc64-softmmu/accel/tcg/translate-all.o
  CC      or1k-softmmu/target/openrisc/mmu.o
  GEN     or1k-softmmu/target/openrisc/decode.inc.c
  CC      or1k-softmmu/target/openrisc/exception_helper.o
  CC      or1k-softmmu/target/openrisc/fpu_helper.o
  CC      ppc64-softmmu/accel/tcg/translator.o
  CC      or1k-softmmu/target/openrisc/interrupt_helper.o
  CC      or1k-softmmu/target/openrisc/mmu_helper.o
  CC      or1k-softmmu/target/openrisc/sys_helper.o
  CC      ppc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      or1k-softmmu/target/openrisc/gdbstub.o
  CC      ppc64-softmmu/hw/block/virtio-blk.o
  GEN     trace/generated-helpers.c
  CC      or1k-softmmu/trace/control-target.o
  CC      or1k-softmmu/target/openrisc/translate.o
  CC      ppc64-softmmu/hw/block/vhost-user-blk.o
  CC      ppc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/char/spapr_vty.o
  CC      ppc64-softmmu/hw/char/virtio-serial-bus.o
  CC      or1k-softmmu/trace/generated-helpers.o
  LINK    or1k-softmmu/qemu-system-or1k
  CC      ppc64-softmmu/hw/core/generic-loader.o
  CC      ppc64-softmmu/hw/core/null-machine.o
  CC      ppc64-softmmu/hw/display/sm501.o
  CC      ppc64-softmmu/hw/display/vga.o
  CC      mips-softmmu/hw/virtio/virtio-crypto.o
  GEN     ppcemb-softmmu/hmp-commands.h
  GEN     ppcemb-softmmu/hmp-commands-info.h
  GEN     ppcemb-softmmu/config-target.h
  CC      ppcemb-softmmu/exec.o
  CC      ppc64-softmmu/hw/display/virtio-gpu.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppcemb-softmmu/tcg/tcg.o
  CC      ppc64-softmmu/hw/display/virtio-vga.o
  CC      ppc64-softmmu/hw/i2c/ppc4xx_i2c.o
  CC      ppc64-softmmu/hw/intc/xics.o
  CC      ppc64-softmmu/hw/intc/xics_spapr.o
  CC      ppc64-softmmu/hw/intc/xics_pnv.o
  CC      ppc64-softmmu/hw/misc/ivshmem.o
  CC      ppcemb-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-softmmu/hw/net/spapr_llan.o
  CC      ppc64-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/net/virtio-net.o
  CC      ppc64-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/registers.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/rings.o
  CC      ppcemb-softmmu/tcg/tcg-op-vec.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppc64-softmmu/hw/nvram/spapr_nvram.o
  CC      ppcemb-softmmu/tcg/tcg-op-gvec.o
  CC      ppc64-softmmu/hw/pci-host/pnv_phb3.o
  CC      ppc64-softmmu/hw/pci-host/pnv_phb3_pbcq.o
  CC      ppc64-softmmu/hw/pci-host/pnv_phb3_msi.o
/var/tmp/patchew-tester-tmp-yhgce4ah/src/hw/pci-host/pnv_phb3_msi.c: In function ‘phb3_msi_reset’:
/var/tmp/patchew-tester-tmp-yhgce4ah/src/hw/pci-host/pnv_phb3_msi.c:230:11: error: ‘ICSStateClass {aka struct ICSStateClass}’ has no member named ‘parent_reset’; did you mean ‘parent_class’?
     icsc->parent_reset(dev);
           ^~~~~~~~~~~~
           parent_class
/var/tmp/patchew-tester-tmp-yhgce4ah/src/hw/pci-host/pnv_phb3_msi.c: In function ‘phb3_msi_realize’:
/var/tmp/patchew-tester-tmp-yhgce4ah/src/hw/pci-host/pnv_phb3_msi.c:261:11: error: ‘ICSStateClass {aka struct ICSStateClass}’ has no member named ‘parent_realize’; did you mean ‘parent_class’?
     icsc->parent_realize(dev, &local_err);
           ^~~~~~~~~~~~~~
           parent_class
/var/tmp/patchew-tester-tmp-yhgce4ah/src/hw/pci-host/pnv_phb3_msi.c: In function ‘phb3_msi_class_init’:
/var/tmp/patchew-tester-tmp-yhgce4ah/src/hw/pci-host/pnv_phb3_msi.c:294:43: error: ‘ICSStateClass {aka struct ICSStateClass}’ has no member named ‘parent_realize’; did you mean ‘parent_class’?
                                     &isc->parent_realize);
                                           ^~~~~~~~~~~~~~
                                           parent_class
/var/tmp/patchew-tester-tmp-yhgce4ah/src/hw/pci-host/pnv_phb3_msi.c:296:41: error: ‘ICSStateClass {aka struct ICSStateClass}’ has no member named ‘parent_reset’; did you mean ‘parent_class’?
                                   &isc->parent_reset);
                                         ^~~~~~~~~~~~
                                         parent_class
make[1]: *** [/var/tmp/patchew-tester-tmp-yhgce4ah/src/rules.mak:69: hw/pci-host/pnv_phb3_msi.o] Error 1
make: *** [Makefile:481: subdir-ppc64-softmmu] Error 2
make: *** Waiting for unfinished jobs....
  CC      ppcemb-softmmu/tcg/tcg-common.o
  CC      mips-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      ppcemb-softmmu/tcg/optimize.o
  CC      mips-softmmu/hw/virtio/vhost.o
  CC      mips-softmmu/hw/virtio/vhost-backend.o
  CC      nios2-softmmu/tcg/tcg.o
  CC      mips-softmmu/hw/virtio/vhost-user.o
  CC      mips-softmmu/hw/virtio/vhost-vsock.o
  CC      mips-softmmu/hw/mips/mips_r4k.o
  CC      mips-softmmu/hw/mips/mips_malta.o
  CC      nios2-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/hw/mips/mips_mipssim.o
  CC      mips-softmmu/hw/mips/addr.o
  CC      mips-softmmu/hw/mips/mips_int.o
  CC      mips-softmmu/hw/mips/gt64xxx_pci.o
  CC      mips-softmmu/hw/mips/cps.o
  CC      mips-softmmu/target/mips/translate.o
  CC      mips-softmmu/target/mips/dsp_helper.o
  CC      nios2-softmmu/tcg/tcg-op-vec.o
  CC      nios2-softmmu/tcg/tcg-op-gvec.o
  CC      nios2-softmmu/tcg/tcg-common.o
  CC      mips-softmmu/target/mips/op_helper.o
  CC      ppcemb-softmmu/fpu/softfloat.o
  CC      mips-softmmu/target/mips/lmi_helper.o
  CC      nios2-softmmu/tcg/optimize.o
  CC      mips-softmmu/target/mips/helper.o
  CC      nios2-softmmu/fpu/softfloat.o
  CC      mips-softmmu/target/mips/cpu.o
  CC      mips-softmmu/target/mips/gdbstub.o
  CC      nios2-softmmu/disas.o
  CC      nios2-softmmu/arch_init.o
  CC      mips-softmmu/target/mips/msa_helper.o
  CC      nios2-softmmu/cpus.o
  CC      nios2-softmmu/monitor.o
  CC      mips-softmmu/target/mips/mips-semi.o
  CC      mips-softmmu/target/mips/machine.o
  CC      nios2-softmmu/gdbstub.o
  CC      mips-softmmu/target/mips/cp0_timer.o
  GEN     trace/generated-helpers.c
  CC      nios2-softmmu/balloon.o
  CC      mips-softmmu/trace/control-target.o
  CC      nios2-softmmu/ioport.o
  CC      nios2-softmmu/numa.o
  CC      nios2-softmmu/qtest.o
  CC      nios2-softmmu/memory.o
  CC      ppcemb-softmmu/disas.o
  CC      nios2-softmmu/memory_mapping.o
  CC      nios2-softmmu/dump.o
  GEN     ppcemb-softmmu/gdbstub-xml.c
  CC      ppcemb-softmmu/arch_init.o
  CC      ppcemb-softmmu/cpus.o
  CC      nios2-softmmu/migration/ram.o
  CC      nios2-softmmu/accel/accel.o
  CC      nios2-softmmu/accel/stubs/hax-stub.o
  CC      nios2-softmmu/accel/stubs/hvf-stub.o
  CC      nios2-softmmu/accel/stubs/whpx-stub.o
  CC      nios2-softmmu/accel/stubs/kvm-stub.o
  CC      nios2-softmmu/accel/tcg/tcg-all.o
  CC      nios2-softmmu/accel/tcg/cputlb.o
  CC      nios2-softmmu/accel/tcg/tcg-runtime.o
  CC      nios2-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      nios2-softmmu/accel/tcg/cpu-exec.o
  CC      mips-softmmu/trace/generated-helpers.o
  LINK    mips-softmmu/qemu-system-mips
  CC      ppcemb-softmmu/monitor.o
  CC      nios2-softmmu/accel/tcg/cpu-exec-common.o
  CC      nios2-softmmu/accel/tcg/translate-all.o
  CC      nios2-softmmu/accel/tcg/translator.o
  CC      ppcemb-softmmu/gdbstub.o
  CC      nios2-softmmu/hw/core/generic-loader.o
  CC      nios2-softmmu/hw/core/null-machine.o
  CC      nios2-softmmu/hw/intc/nios2_iic.o
  CC      nios2-softmmu/hw/misc/mmio_interface.o
  CC      nios2-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/balloon.o
  CC      nios2-softmmu/hw/net/rocker/qmp-norocker.o
  CC      nios2-softmmu/hw/timer/altera_timer.o
  CC      nios2-softmmu/hw/vfio/common.o
  CC      ppcemb-softmmu/ioport.o
  CC      nios2-softmmu/hw/vfio/platform.o
  CC      nios2-softmmu/hw/vfio/spapr.o
  CC      ppcemb-softmmu/numa.o
  CC      nios2-softmmu/hw/nios2/boot.o
  CC      nios2-softmmu/hw/nios2/cpu_pic.o
  CC      nios2-softmmu/hw/nios2/10m50_devboard.o
  CC      nios2-softmmu/target/nios2/translate.o
  CC      ppcemb-softmmu/qtest.o
  CC      nios2-softmmu/target/nios2/op_helper.o
  CC      nios2-softmmu/target/nios2/helper.o
  CC      nios2-softmmu/target/nios2/cpu.o
  CC      ppcemb-softmmu/memory.o
  CC      ppcemb-softmmu/memory_mapping.o
  CC      ppcemb-softmmu/dump.o
  CC      ppcemb-softmmu/migration/ram.o
  CC      ppcemb-softmmu/accel/accel.o
  CC      ppcemb-softmmu/accel/stubs/hax-stub.o
  CC      ppcemb-softmmu/accel/stubs/hvf-stub.o
  CC      ppcemb-softmmu/accel/stubs/whpx-stub.o
  CC      ppcemb-softmmu/accel/stubs/kvm-stub.o
  CC      ppcemb-softmmu/accel/tcg/tcg-all.o
  CC      ppcemb-softmmu/accel/tcg/cputlb.o
  CC      ppcemb-softmmu/accel/tcg/tcg-runtime.o
  CC      ppcemb-softmmu/accel/tcg/tcg-runtime-gvec.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppcemb-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/accel/tcg/translator.o
  CC      ppcemb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppcemb-softmmu/hw/block/virtio-blk.o
  CC      ppcemb-softmmu/hw/block/vhost-user-blk.o
  CC      ppcemb-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppcemb-softmmu/hw/char/virtio-serial-bus.o
  CC      ppcemb-softmmu/hw/core/generic-loader.o
  CC      ppcemb-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/display/sm501.o
  CC      ppcemb-softmmu/hw/display/vga.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppcemb-softmmu/hw/i2c/ppc4xx_i2c.o
  CC      ppcemb-softmmu/hw/misc/ivshmem.o
  CC      ppcemb-softmmu/hw/misc/mmio_interface.o
  CC      ppcemb-softmmu/hw/net/xilinx_ethlite.o
  CC      ppcemb-softmmu/hw/net/virtio-net.o
  CC      ppcemb-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi.o
  CC      ppcemb-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppcemb-softmmu/hw/vfio/common.o
  CC      ppcemb-softmmu/hw/vfio/pci.o
  CC      ppcemb-softmmu/hw/vfio/pci-quirks.o
  CC      ppcemb-softmmu/hw/vfio/display.o
  CC      ppcemb-softmmu/hw/vfio/platform.o
  CC      ppcemb-softmmu/hw/vfio/spapr.o
  CC      ppcemb-softmmu/hw/virtio/virtio.o
  CC      ppcemb-softmmu/hw/virtio/virtio-balloon.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      ppcemb-softmmu/hw/virtio/vhost.o
  CC      ppcemb-softmmu/hw/virtio/vhost-backend.o
  CC      ppcemb-softmmu/hw/virtio/vhost-user.o
  CC      ppcemb-softmmu/hw/virtio/vhost-vsock.o
  CC      ppcemb-softmmu/hw/ppc/ppc.o
  CC      ppcemb-softmmu/hw/ppc/ppc_booke.o
  CC      ppcemb-softmmu/hw/ppc/fdt.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_devs.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_uc.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_boards.o
  CC      ppcemb-softmmu/hw/ppc/ppc440_bamboo.o
  CC      ppcemb-softmmu/hw/ppc/ppc440_pcix.o
  CC      ppcemb-softmmu/hw/ppc/ppc440_uc.o
  CC      ppcemb-softmmu/hw/ppc/sam460ex.o
  CC      ppcemb-softmmu/hw/ppc/virtex_ml507.o
  CC      ppcemb-softmmu/target/ppc/cpu-models.o
  CC      ppcemb-softmmu/target/ppc/cpu.o
  CC      ppcemb-softmmu/target/ppc/translate.o
  CC      ppcemb-softmmu/target/ppc/machine.o
  CC      ppcemb-softmmu/target/ppc/mmu_helper.o
  CC      ppcemb-softmmu/target/ppc/mmu-hash32.o
  CC      ppcemb-softmmu/target/ppc/monitor.o
  CC      ppcemb-softmmu/target/ppc/arch_dump.o
  CC      ppcemb-softmmu/target/ppc/kvm-stub.o
  CC      ppcemb-softmmu/target/ppc/dfp_helper.o
  CC      ppcemb-softmmu/target/ppc/excp_helper.o
  CC      ppcemb-softmmu/target/ppc/fpu_helper.o
  CC      ppcemb-softmmu/target/ppc/int_helper.o
  CC      ppcemb-softmmu/target/ppc/timebase_helper.o
  CC      ppcemb-softmmu/target/ppc/misc_helper.o
  CC      ppcemb-softmmu/target/ppc/mem_helper.o
  CC      ppcemb-softmmu/target/ppc/gdbstub.o
  CC      ppcemb-softmmu/target/ppc/../../libdecnumber/decContext.o
  CC      ppcemb-softmmu/target/ppc/../../libdecnumber/decNumber.o
  CC      ppcemb-softmmu/target/ppc/../../libdecnumber/dpd/decimal32.o
  CC      ppcemb-softmmu/target/ppc/../../libdecnumber/dpd/decimal64.o
  CC      ppcemb-softmmu/target/ppc/../../libdecnumber/dpd/decimal128.o
  GEN     trace/generated-helpers.c
  CC      ppcemb-softmmu/trace/control-target.o
  CC      ppcemb-softmmu/gdbstub-xml.o
  CC      ppcemb-softmmu/trace/generated-helpers.o
  CC      nios2-softmmu/target/nios2/monitor.o
  GEN     trace/generated-helpers.c
  CC      nios2-softmmu/target/nios2/mmu.o
  CC      nios2-softmmu/trace/control-target.o
  CC      nios2-softmmu/trace/generated-helpers.o
  LINK    ppcemb-softmmu/qemu-system-ppcemb
  LINK    nios2-softmmu/qemu-system-nios2
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 1/2] " Cédric Le Goater
@ 2018-07-09  7:22   ` Cédric Le Goater
  2018-07-18  6:13     ` David Gibson
  2018-07-18  6:12   ` David Gibson
  2018-07-26  9:03   ` Cédric Le Goater
  2 siblings, 1 reply; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-09  7:22 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

On 06/28/2018 10:36 AM, Cédric Le Goater wrote:
> From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> 
> This is a model of the PCIe Host Bridge (PHB3) found on a Power8
> processor. It includes the PowerBus logic interface (PBCQ), IOMMU
> support, PCIe root complex, the XICS MSI and LSI interrupt sources.
> 
> The Power8 processor comes in different flavors: Venice, Murano,
> Naple, each having a different number of PHBs. All are initialized but
> the machine only needs one PCI bus to plug a couple of devices (net,
> storage, usbs). The other busses are provided to test complex
> configurations. Some platforms, like the Firestone, can also couple
> PHBs on the first chip to provide more bandwidth but this is too
> specific to model in QEMU.
> 
> No default device layout is provided and PCI devices can be added on
> any of the available PCI busses (pci.0..2 on a Power8 chip) using
> command line options such as :
> 
>   -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.0,addr=0x2
>   -netdev bridge,id=net0,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=hostnet0
> 
>   -device megasas,id=scsi0,bus=pci.0,addr=0x1
>   -drive file=$disk,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none
>   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2
> 
> Multi chip is supported, each chip adding its set of PHB3 controllers
> and its PCI busses. The model doesn't emulate the EEH error handling
> (and may never do).
Some comments,

The model should initialize *two* default PHBs per chip for all 
processor flavors. Extra ones would come with user creatable PHBs,
which can be added later.

Can someone take a closer look at pnv_phb3_pci_create() ? This is 
where the PCI busses, bridges, etc are created.

PBCQ is a logic unit acting as a bridge on the PowerBus but the model 
might be adding some extra unnecessary complexity. may be the PHB3 and 
the PBCQ models could be merged. This is minor I think.

In which sub-tree should this patchset be merged ? PCI or PPC ?

Thanks,

C.  
 

> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> [clg: rewrote the QOM models
>       misc fixes
>       lots of love and care]
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
>  default-configs/ppc64-softmmu.mak   |    1 +
>  include/hw/pci-host/pnv_phb3.h      |  161 +++++
>  include/hw/pci-host/pnv_phb3_regs.h |  467 ++++++++++++++
>  include/hw/ppc/pnv.h                |   22 +
>  include/hw/ppc/pnv_xscom.h          |    9 +
>  include/hw/ppc/xics.h               |    1 +
>  hw/intc/xics.c                      |    2 +-
>  hw/pci-host/pnv_phb3.c              | 1219 +++++++++++++++++++++++++++++++++++
>  hw/pci-host/pnv_phb3_msi.c          |  316 +++++++++
>  hw/pci-host/pnv_phb3_pbcq.c         |  347 ++++++++++
>  hw/ppc/pnv.c                        |   75 ++-
>  hw/ppc/pnv_xscom.c                  |    6 +-
>  hw/pci-host/Makefile.objs           |    1 +
>  13 files changed, 2622 insertions(+), 5 deletions(-)
>  create mode 100644 include/hw/pci-host/pnv_phb3.h
>  create mode 100644 include/hw/pci-host/pnv_phb3_regs.h
>  create mode 100644 hw/pci-host/pnv_phb3.c
>  create mode 100644 hw/pci-host/pnv_phb3_msi.c
>  create mode 100644 hw/pci-host/pnv_phb3_pbcq.c
> 
> diff --git a/default-configs/ppc64-softmmu.mak b/default-configs/ppc64-softmmu.mak
> index b94af6c7c62a..deebba2b044a 100644
> --- a/default-configs/ppc64-softmmu.mak
> +++ b/default-configs/ppc64-softmmu.mak
> @@ -9,6 +9,7 @@ CONFIG_IPMI=y
>  CONFIG_IPMI_LOCAL=y
>  CONFIG_IPMI_EXTERN=y
>  CONFIG_ISA_IPMI_BT=y
> +CONFIG_PCIE_PORT=y
>  
>  # For pSeries
>  CONFIG_PSERIES=y
> diff --git a/include/hw/pci-host/pnv_phb3.h b/include/hw/pci-host/pnv_phb3.h
> new file mode 100644
> index 000000000000..459994b751ca
> --- /dev/null
> +++ b/include/hw/pci-host/pnv_phb3.h
> @@ -0,0 +1,161 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +
> +#ifndef PCI_HOST_PNV_PHB3_H
> +#define PCI_HOST_PNV_PHB3_H
> +
> +#include "hw/pci/pci_host.h"
> +#include "hw/ppc/xics.h"
> +
> +typedef struct PnvPHB3 PnvPHB3;
> +typedef struct PnvChip PnvChip;
> +
> +/*
> + * PHB3 XICS Source for MSIs
> + */
> +#define TYPE_PHB3_MSI "phb3-msi"
> +#define PHB3_MSI(obj) OBJECT_CHECK(Phb3MsiState, (obj), TYPE_PHB3_MSI)
> +
> +#define PHB3_MAX_MSI     2048
> +
> +typedef struct Phb3MsiState {
> +    ICSState ics;
> +
> +    PnvPHB3 *phb;
> +    uint64_t rba[PHB3_MAX_MSI / 64];
> +    uint32_t rba_sum;
> +} Phb3MsiState;
> +
> +void pnv_phb3_msi_update_config(Phb3MsiState *msis, uint32_t base,
> +                                uint32_t count);
> +void pnv_phb3_msi_send(Phb3MsiState *msis, uint64_t addr, uint16_t data,
> +                       int32_t dev_pe);
> +void pnv_phb3_msi_ffi(Phb3MsiState *msis, uint64_t val);
> +
> +
> +/* We have one such address space wrapper per possible device
> + * under the PHB since they need to be assigned statically at
> + * qemu device creation time. The relationship to a PE is done
> + * later dynamically. This means we can potentially create a lot
> + * of these guys. Q35 stores them as some kind of radix tree but
> + * we never really need to do fast lookups so instead we simply
> + * keep a QLIST of them for now, we can add the radix if needed
> + * later on.
> + *
> + * We do cache the PE number to speed things up a bit though.
> + */
> +typedef struct PnvPhb3DMASpace {
> +    PCIBus *bus;
> +    uint8_t devfn;
> +    int pe_num;         /* Cached PE number */
> +#define PHB_INVALID_PE (-1)
> +    PnvPHB3 *phb;
> +    AddressSpace dma_as;
> +    IOMMUMemoryRegion dma_mr;
> +    MemoryRegion msi32_mr;
> +    MemoryRegion msi64_mr;
> +    bool msi32_mapped;
> +    bool msi64_mapped;
> +    QLIST_ENTRY(PnvPhb3DMASpace) list;
> +} PnvPhb3DMASpace;
> +
> +/*
> + * PHB3 Power Bus Common Queue
> + */
> +#define TYPE_PNV_PBCQ "pnv-pbcq"
> +#define PNV_PBCQ(obj) OBJECT_CHECK(PnvPBCQState, (obj), TYPE_PNV_PBCQ)
> +
> +typedef struct PnvPBCQState {
> +    DeviceState parent;
> +
> +    uint32_t nest_xbase;
> +    uint32_t spci_xbase;
> +    uint32_t pci_xbase;
> +#define PBCQ_NEST_REGS_COUNT    0x46
> +#define PBCQ_PCI_REGS_COUNT     0x15
> +#define PBCQ_SPCI_REGS_COUNT    0x5
> +
> +    uint64_t nest_regs[PBCQ_NEST_REGS_COUNT];
> +    uint64_t spci_regs[PBCQ_SPCI_REGS_COUNT];
> +    uint64_t pci_regs[PBCQ_PCI_REGS_COUNT];
> +    MemoryRegion mmbar0;
> +    MemoryRegion mmbar1;
> +    MemoryRegion phbbar;
> +    bool mmio0_mapped;
> +    bool mmio1_mapped;
> +    bool phb_mapped;
> +    uint64_t mmio0_base;
> +    uint64_t mmio0_size;
> +    uint64_t mmio1_base;
> +    uint64_t mmio1_size;
> +    PnvPHB3 *phb;
> +
> +    MemoryRegion xscom_nest_regs;
> +    MemoryRegion xscom_pci_regs;
> +    MemoryRegion xscom_spci_regs;
> +} PnvPBCQState;
> +
> +/*
> + * PHB3 PCI Host Bridge for PowerNV machines (POWER8)
> + */
> +#define TYPE_PNV_PHB3 "pnv-phb3"
> +#define PNV_PHB3(obj) OBJECT_CHECK(PnvPHB3, (obj), TYPE_PNV_PHB3)
> +
> +#define PNV_PHB3_NUM_M64      16
> +#define PNV_PHB3_NUM_REGS     (0x1000 >> 3)
> +#define PNV_PHB3_NUM_LSI      8
> +#define PNV_PHB3_NUM_PE       256
> +
> +#define PCI_MMIO_TOTAL_SIZE   (0x1ull << 60)
> +
> +struct PnvPHB3 {
> +    PCIHostState parent_obj;
> +
> +    uint32_t chip_id;
> +    uint32_t phb_id;
> +    char bus_path[8];
> +
> +    uint64_t regs[PNV_PHB3_NUM_REGS];
> +    MemoryRegion mr_regs;
> +
> +    MemoryRegion mr_m32;
> +    MemoryRegion mr_m64[PNV_PHB3_NUM_M64];
> +    bool regs_mapped;
> +    bool m32_mapped;
> +    bool m64_mapped[PNV_PHB3_NUM_M64];
> +    MemoryRegion pci_mmio;
> +    MemoryRegion pci_io;
> +
> +    uint64_t ioda_LIST[8];
> +    uint64_t ioda_LXIVT[8];
> +    uint64_t ioda_TVT[512];
> +    uint64_t ioda_M64BT[16];
> +    uint64_t ioda_MDT[256];
> +    uint64_t ioda_PEEV[4];
> +
> +    uint32_t total_irq;
> +    ICSState lsis;
> +    Phb3MsiState msis;
> +
> +    PnvPBCQState pbcq;
> +
> +    QLIST_HEAD(, PnvPhb3DMASpace) dma_spaces;
> +};
> +
> +#define TYPE_PNV_PHB3_RC "pnv-phb3-rc"
> +
> +#define TYPE_PNV_PHB3_ROOT_BUS "pnv-phb3-root-bus"
> +
> +uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size);
> +void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size);
> +void pnv_phb3_update_regions(PnvPHB3 *phb);
> +void pnv_phb3_remap_irqs(PnvPHB3 *phb);
> +
> +
> +#endif /* PCI_HOST_PNV_PHB3_H */
> diff --git a/include/hw/pci-host/pnv_phb3_regs.h b/include/hw/pci-host/pnv_phb3_regs.h
> new file mode 100644
> index 000000000000..4ddbb5559eb5
> --- /dev/null
> +++ b/include/hw/pci-host/pnv_phb3_regs.h
> @@ -0,0 +1,467 @@
> +/* Copyright (c) 2013-2018, IBM Corporation.
> + *
> + * Licensed under the Apache License, Version 2.0 (the "License");
> + * you may not use this file except in compliance with the License.
> + * You may obtain a copy of the License at
> + *
> + *      http://www.apache.org/licenses/LICENSE-2.0
> + *
> + * Unless required by applicable law or agreed to in writing, software
> + * distributed under the License is distributed on an "AS IS" BASIS,
> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> + * implied.
> + *
> + * See the License for the specific language governing permissions and
> + * limitations under the License.
> + */
> +
> +#ifndef PCI_HOST_PNV_PHB3_REGS_H
> +#define PCI_HOST_PNV_PHB3_REGS_H
> +
> +/*
> + * PBCQ XSCOM registers
> + */
> +
> +#define PBCQ_NEST_IRSN_COMPARE  0x1a
> +#define PBCQ_NEST_IRSN_COMP           PPC_BITMASK(0, 18)
> +#define PBCQ_NEST_IRSN_MASK     0x1b
> +#define PBCQ_NEST_LSI_SRC_ID    0x1f
> +#define   PBCQ_NEST_LSI_SRC           PPC_BITMASK(0, 7)
> +#define PBCQ_NEST_REGS_COUNT    0x46
> +#define PBCQ_NEST_MMIO_BAR0     0x40
> +#define PBCQ_NEST_MMIO_BAR1     0x41
> +#define PBCQ_NEST_PHB_BAR       0x42
> +#define PBCQ_NEST_MMIO_MASK0    0x43
> +#define PBCQ_NEST_MMIO_MASK1    0x44
> +#define PBCQ_NEST_BAR_EN        0x45
> +#define   PBCQ_NEST_BAR_EN_MMIO0    PPC_BIT(0)
> +#define   PBCQ_NEST_BAR_EN_MMIO1    PPC_BIT(1)
> +#define   PBCQ_NEST_BAR_EN_PHB      PPC_BIT(2)
> +#define   PBCQ_NEST_BAR_EN_IRSN_RX  PPC_BIT(3)
> +#define   PBCQ_NEST_BAR_EN_IRSN_TX  PPC_BIT(4)
> +
> +#define PBCQ_PCI_REGS_COUNT     0x15
> +#define PBCQ_PCI_BAR2           0x0b
> +
> +#define PBCQ_SPCI_REGS_COUNT    0x5
> +#define PBCQ_SPCI_ASB_ADDR      0x0
> +#define PBCQ_SPCI_ASB_STATUS    0x1
> +#define PBCQ_SPCI_ASB_DATA      0x2
> +#define PBCQ_SPCI_AIB_CAPP_EN   0x3
> +#define PBCQ_SPCI_CAPP_SEC_TMR  0x4
> +
> +/*
> + * PHB MMIO registers
> + */
> +
> +/* PHB Fundamental register set A */
> +#define PHB_LSI_SOURCE_ID               0x100
> +#define   PHB_LSI_SRC_ID                PPC_BITMASK(5, 12)
> +#define PHB_DMA_CHAN_STATUS             0x110
> +#define   PHB_DMA_CHAN_ANY_ERR          PPC_BIT(27)
> +#define   PHB_DMA_CHAN_ANY_ERR1         PPC_BIT(28)
> +#define   PHB_DMA_CHAN_ANY_FREEZE       PPC_BIT(29)
> +#define PHB_CPU_LOADSTORE_STATUS        0x120
> +#define   PHB_CPU_LS_ANY_ERR            PPC_BIT(27)
> +#define   PHB_CPU_LS_ANY_ERR1           PPC_BIT(28)
> +#define   PHB_CPU_LS_ANY_FREEZE         PPC_BIT(29)
> +#define PHB_DMA_MSI_NODE_ID             0x128
> +#define   PHB_DMAMSI_NID_FIXED          PPC_BIT(0)
> +#define   PHB_DMAMSI_NID_MASK           PPC_BITMASK(24, 31)
> +#define   PHB_DMAMSI_NID_LSH            PPC_BITLSHIFT(31)
> +#define PHB_CONFIG_DATA                 0x130
> +#define PHB_LOCK0                       0x138
> +#define PHB_CONFIG_ADDRESS              0x140
> +#define   PHB_CA_ENABLE                 PPC_BIT(0)
> +#define   PHB_CA_BUS_MASK               PPC_BITMASK(4, 11)
> +#define   PHB_CA_BUS_LSH                PPC_BITLSHIFT(11)
> +#define   PHB_CA_DEV_MASK               PPC_BITMASK(12, 16)
> +#define   PHB_CA_DEV_LSH                PPC_BITLSHIFT(16)
> +#define   PHB_CA_FUNC_MASK              PPC_BITMASK(17, 19)
> +#define   PHB_CA_FUNC_LSH               PPC_BITLSHIFT(19)
> +#define   PHB_CA_REG_MASK               PPC_BITMASK(20, 31)
> +#define   PHB_CA_REG_LSH                PPC_BITLSHIFT(31)
> +#define   PHB_CA_PE_MASK                PPC_BITMASK(40, 47)
> +#define   PHB_CA_PE_LSH                 PPC_BITLSHIFT(47)
> +#define PHB_LOCK1                       0x148
> +#define PHB_IVT_BAR                     0x150
> +#define   PHB_IVT_BAR_ENABLE            PPC_BIT(0)
> +#define   PHB_IVT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 48)
> +#define   PHB_IVT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(48)
> +#define   PHB_IVT_LENGTH_MASK           PPC_BITMASK(52, 63)
> +#define   PHB_IVT_LENGTH_ADDRESS_LSH    PPC_BITLSHIFT(63)
> +#define PHB_RBA_BAR                     0x158
> +#define   PHB_RBA_BAR_ENABLE            PPC_BIT(0)
> +#define   PHB_RBA_BASE_ADDRESS_MASK     PPC_BITMASK(14, 55)
> +#define   PHB_RBA_BASE_ADDRESS_LSH      PPC_BITLSHIFT(55)
> +#define PHB_PHB3_CONFIG                 0x160
> +#define   PHB_PHB3C_64B_TCE_EN          PPC_BIT(2)
> +#define   PHB_PHB3C_32BIT_MSI_EN        PPC_BIT(8)
> +#define   PHB_PHB3C_64BIT_MSI_EN        PPC_BIT(14)
> +#define   PHB_PHB3C_M32_EN              PPC_BIT(16)
> +#define PHB_RTT_BAR                     0x168
> +#define   PHB_RTT_BAR_ENABLE            PPC_BIT(0)
> +#define   PHB_RTT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 46)
> +#define   PHB_RTT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(46)
> +#define PHB_PELTV_BAR                   0x188
> +#define   PHB_PELTV_BAR_ENABLE          PPC_BIT(0)
> +#define   PHB_PELTV_BASE_ADDRESS_MASK   PPC_BITMASK(14, 50)
> +#define   PHB_PELTV_BASE_ADDRESS_LSH    PPC_BITLSHIFT(50)
> +#define PHB_M32_BASE_ADDR               0x190
> +#define PHB_M32_BASE_MASK               0x198
> +#define PHB_M32_START_ADDR              0x1a0
> +#define PHB_PEST_BAR                    0x1a8
> +#define   PHB_PEST_BAR_ENABLE           PPC_BIT(0)
> +#define   PHB_PEST_BASE_ADDRESS_MASK    PPC_BITMASK(14, 51)
> +#define   PHB_PEST_BASE_ADDRESS_LSH     PPC_BITLSHIFT(51)
> +#define PHB_M64_UPPER_BITS              0x1f0
> +#define PHB_INTREP_TIMER                0x1f8
> +#define PHB_DMARD_SYNC                  0x200
> +#define PHB_RTC_INVALIDATE              0x208
> +#define   PHB_RTC_INVALIDATE_ALL        PPC_BIT(0)
> +#define   PHB_RTC_INVALIDATE_RID_MASK   PPC_BITMASK(16, 31)
> +#define   PHB_RTC_INVALIDATE_RID_LSH    PPC_BITLSHIFT(31)
> +#define PHB_TCE_KILL                    0x210
> +#define   PHB_TCE_KILL_ALL              PPC_BIT(0)
> +#define PHB_TCE_SPEC_CTL                0x218
> +#define PHB_IODA_ADDR                   0x220
> +#define   PHB_IODA_AD_AUTOINC           PPC_BIT(0)
> +#define   PHB_IODA_AD_TSEL              PPC_BITMASK(11, 15)
> +#define   PHB_IODA_AD_TADR              PPC_BITMASK(55, 63)
> +#define PHB_IODA_DATA0                  0x228
> +#define PHB_FFI_REQUEST                 0x238
> +#define   PHB_FFI_LOCK_CLEAR            PPC_BIT(3)
> +#define   PHB_FFI_REQUEST_ISN_MASK      PPC_BITMASK(49, 59)
> +#define   PHB_FFI_REQUEST_ISN_LSH       PPC_BITLSHIFT(59)
> +#define PHB_FFI_LOCK                    0x240
> +#define   PHB_FFI_LOCK_STATE            PPC_BIT(0)
> +#define PHB_XIVE_UPDATE                 0x248 /* Broken in DD1 */
> +#define PHB_PHB3_GEN_CAP                0x250
> +#define PHB_PHB3_TCE_CAP                0x258
> +#define PHB_PHB3_IRQ_CAP                0x260
> +#define PHB_PHB3_EEH_CAP                0x268
> +#define PHB_IVC_INVALIDATE              0x2a0
> +#define   PHB_IVC_INVALIDATE_ALL        PPC_BIT(0)
> +#define   PHB_IVC_INVALIDATE_SID_MASK   PPC_BITMASK(16, 31)
> +#define   PHB_IVC_INVALIDATE_SID_LSH    PPC_BITLSHIFT(31)
> +#define PHB_IVC_UPDATE                  0x2a8
> +#define   PHB_IVC_UPDATE_ENABLE_P       PPC_BIT(0)
> +#define   PHB_IVC_UPDATE_ENABLE_Q       PPC_BIT(1)
> +#define   PHB_IVC_UPDATE_ENABLE_SERVER  PPC_BIT(2)
> +#define   PHB_IVC_UPDATE_ENABLE_PRI     PPC_BIT(3)
> +#define   PHB_IVC_UPDATE_ENABLE_GEN     PPC_BIT(4)
> +#define   PHB_IVC_UPDATE_ENABLE_CON     PPC_BIT(5)
> +#define   PHB_IVC_UPDATE_GEN_MATCH_MASK PPC_BITMASK(6, 7)
> +#define   PHB_IVC_UPDATE_GEN_MATCH_LSH  PPC_BITLSHIFT(7)
> +#define   PHB_IVC_UPDATE_SERVER_MASK    PPC_BITMASK(8, 23)
> +#define   PHB_IVC_UPDATE_SERVER_LSH     PPC_BITLSHIFT(23)
> +#define   PHB_IVC_UPDATE_PRI_MASK       PPC_BITMASK(24, 31)
> +#define   PHB_IVC_UPDATE_PRI_LSH        PPC_BITLSHIFT(31)
> +#define   PHB_IVC_UPDATE_GEN_MASK       PPC_BITMASK(32, 33)
> +#define   PHB_IVC_UPDATE_GEN_LSH        PPC_BITLSHIFT(33)
> +#define   PHB_IVC_UPDATE_P_MASK         PPC_BITMASK(34, 34)
> +#define   PHB_IVC_UPDATE_P_LSH          PPC_BITLSHIFT(34)
> +#define   PHB_IVC_UPDATE_Q_MASK         PPC_BITMASK(35, 35)
> +#define   PHB_IVC_UPDATE_Q_LSH          PPC_BITLSHIFT(35)
> +#define   PHB_IVC_UPDATE_SID_MASK       PPC_BITMASK(48, 63)
> +#define   PHB_IVC_UPDATE_SID_LSH        PPC_BITLSHIFT(63)
> +#define PHB_PAPR_ERR_INJ_CTL            0x2b0
> +#define   PHB_PAPR_ERR_INJ_CTL_INB      PPC_BIT(0)
> +#define   PHB_PAPR_ERR_INJ_CTL_OUTB     PPC_BIT(1)
> +#define   PHB_PAPR_ERR_INJ_CTL_STICKY   PPC_BIT(2)
> +#define   PHB_PAPR_ERR_INJ_CTL_CFG      PPC_BIT(3)
> +#define   PHB_PAPR_ERR_INJ_CTL_RD       PPC_BIT(4)
> +#define   PHB_PAPR_ERR_INJ_CTL_WR       PPC_BIT(5)
> +#define   PHB_PAPR_ERR_INJ_CTL_FREEZE   PPC_BIT(6)
> +#define PHB_PAPR_ERR_INJ_ADDR           0x2b8
> +#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_MASK       PPC_BITMASK(16, 63)
> +#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_LSH        PPC_BITLSHIFT(63)
> +#define PHB_PAPR_ERR_INJ_MASK           0x2c0
> +#define   PHB_PAPR_ERR_INJ_MASK_CFG_MASK        PPC_BITMASK(4, 11)
> +#define   PHB_PAPR_ERR_INJ_MASK_CFG_LSH         PPC_BITLSHIFT(11)
> +#define   PHB_PAPR_ERR_INJ_MASK_MMIO_MASK       PPC_BITMASK(16, 63)
> +#define   PHB_PAPR_ERR_INJ_MASK_MMIO_LSH        PPC_BITLSHIFT(63)
> +#define PHB_ETU_ERR_SUMMARY             0x2c8
> +
> +/*  UTL registers */
> +#define UTL_SYS_BUS_CONTROL             0x400
> +#define UTL_STATUS                      0x408
> +#define UTL_SYS_BUS_AGENT_STATUS        0x410
> +#define UTL_SYS_BUS_AGENT_ERR_SEVERITY  0x418
> +#define UTL_SYS_BUS_AGENT_IRQ_EN        0x420
> +#define UTL_SYS_BUS_BURST_SZ_CONF       0x440
> +#define UTL_REVISION_ID                 0x448
> +#define UTL_BCLK_DOMAIN_DBG1            0x460
> +#define UTL_BCLK_DOMAIN_DBG2            0x468
> +#define UTL_BCLK_DOMAIN_DBG3            0x470
> +#define UTL_BCLK_DOMAIN_DBG4            0x478
> +#define UTL_BCLK_DOMAIN_DBG5            0x480
> +#define UTL_BCLK_DOMAIN_DBG6            0x488
> +#define UTL_OUT_POST_HDR_BUF_ALLOC      0x4c0
> +#define UTL_OUT_POST_DAT_BUF_ALLOC      0x4d0
> +#define UTL_IN_POST_HDR_BUF_ALLOC       0x4e0
> +#define UTL_IN_POST_DAT_BUF_ALLOC       0x4f0
> +#define UTL_OUT_NP_BUF_ALLOC            0x500
> +#define UTL_IN_NP_BUF_ALLOC             0x510
> +#define UTL_PCIE_TAGS_ALLOC             0x520
> +#define UTL_GBIF_READ_TAGS_ALLOC        0x530
> +#define UTL_PCIE_PORT_CONTROL           0x540
> +#define UTL_PCIE_PORT_STATUS            0x548
> +#define UTL_PCIE_PORT_ERROR_SEV         0x550
> +#define UTL_PCIE_PORT_IRQ_EN            0x558
> +#define UTL_RC_STATUS                   0x560
> +#define UTL_RC_ERR_SEVERITY             0x568
> +#define UTL_RC_IRQ_EN                   0x570
> +#define UTL_EP_STATUS                   0x578
> +#define UTL_EP_ERR_SEVERITY             0x580
> +#define UTL_EP_ERR_IRQ_EN               0x588
> +#define UTL_PCI_PM_CTRL1                0x590
> +#define UTL_PCI_PM_CTRL2                0x598
> +#define UTL_GP_CTL1                     0x5a0
> +#define UTL_GP_CTL2                     0x5a8
> +#define UTL_PCLK_DOMAIN_DBG1            0x5b0
> +#define UTL_PCLK_DOMAIN_DBG2            0x5b8
> +#define UTL_PCLK_DOMAIN_DBG3            0x5c0
> +#define UTL_PCLK_DOMAIN_DBG4            0x5c8
> +
> +/* PCI-E Stack registers */
> +#define PHB_PCIE_SYSTEM_CONFIG          0x600
> +#define PHB_PCIE_BUS_NUMBER             0x608
> +#define PHB_PCIE_SYSTEM_TEST            0x618
> +#define PHB_PCIE_LINK_MANAGEMENT        0x630
> +#define   PHB_PCIE_LM_LINK_ACTIVE       PPC_BIT(8)
> +#define PHB_PCIE_DLP_TRAIN_CTL          0x640
> +#define   PHB_PCIE_DLP_TCTX_DISABLE     PPC_BIT(1)
> +#define   PHB_PCIE_DLP_TCRX_DISABLED    PPC_BIT(16)
> +#define   PHB_PCIE_DLP_INBAND_PRESENCE  PPC_BIT(19)
> +#define   PHB_PCIE_DLP_TC_DL_LINKUP     PPC_BIT(21)
> +#define   PHB_PCIE_DLP_TC_DL_PGRESET    PPC_BIT(22)
> +#define   PHB_PCIE_DLP_TC_DL_LINKACT    PPC_BIT(23)
> +#define PHB_PCIE_SLOP_LOOPBACK_STATUS   0x648
> +#define PHB_PCIE_SYS_LINK_INIT          0x668
> +#define PHB_PCIE_UTL_CONFIG             0x670
> +#define PHB_PCIE_DLP_CONTROL            0x678
> +#define PHB_PCIE_UTL_ERRLOG1            0x680
> +#define PHB_PCIE_UTL_ERRLOG2            0x688
> +#define PHB_PCIE_UTL_ERRLOG3            0x690
> +#define PHB_PCIE_UTL_ERRLOG4            0x698
> +#define PHB_PCIE_DLP_ERRLOG1            0x6a0
> +#define PHB_PCIE_DLP_ERRLOG2            0x6a8
> +#define PHB_PCIE_DLP_ERR_STATUS         0x6b0
> +#define PHB_PCIE_DLP_ERR_COUNTERS       0x6b8
> +#define PHB_PCIE_UTL_ERR_INJECT         0x6c0
> +#define PHB_PCIE_TLDLP_ERR_INJECT       0x6c8
> +#define PHB_PCIE_LANE_EQ_CNTL0          0x6d0
> +#define PHB_PCIE_LANE_EQ_CNTL1          0x6d8
> +#define PHB_PCIE_LANE_EQ_CNTL2          0x6e0
> +#define PHB_PCIE_LANE_EQ_CNTL3          0x6e8
> +#define PHB_PCIE_STRAPPING              0x700
> +
> +/* Fundamental register set B */
> +#define PHB_VERSION                     0x800
> +#define PHB_RESET                       0x808
> +#define PHB_CONTROL                     0x810
> +#define   PHB_CTRL_IVE_128_BYTES        PPC_BIT(24)
> +#define PHB_AIB_RX_CRED_INIT_TIMER      0x818
> +#define PHB_AIB_RX_CMD_CRED             0x820
> +#define PHB_AIB_RX_DATA_CRED            0x828
> +#define PHB_AIB_TX_CMD_CRED             0x830
> +#define PHB_AIB_TX_DATA_CRED            0x838
> +#define PHB_AIB_TX_CHAN_MAPPING         0x840
> +#define PHB_AIB_TAG_ENABLE              0x858
> +#define PHB_AIB_FENCE_CTRL              0x860
> +#define PHB_TCE_TAG_ENABLE              0x868
> +#define PHB_TCE_WATERMARK               0x870
> +#define PHB_TIMEOUT_CTRL1               0x878
> +#define PHB_TIMEOUT_CTRL2               0x880
> +#define PHB_QUIESCE_DMA_G               0x888
> +#define PHB_AIB_TAG_STATUS              0x900
> +#define PHB_TCE_TAG_STATUS              0x908
> +
> +/* FIR & Error registers */
> +#define PHB_LEM_FIR_ACCUM               0xc00
> +#define PHB_LEM_FIR_AND_MASK            0xc08
> +#define PHB_LEM_FIR_OR_MASK             0xc10
> +#define PHB_LEM_ERROR_MASK              0xc18
> +#define PHB_LEM_ERROR_AND_MASK          0xc20
> +#define PHB_LEM_ERROR_OR_MASK           0xc28
> +#define PHB_LEM_ACTION0                 0xc30
> +#define PHB_LEM_ACTION1                 0xc38
> +#define PHB_LEM_WOF                     0xc40
> +#define PHB_ERR_STATUS                  0xc80
> +#define PHB_ERR1_STATUS                 0xc88
> +#define PHB_ERR_INJECT                  0xc90
> +#define PHB_ERR_LEM_ENABLE              0xc98
> +#define PHB_ERR_IRQ_ENABLE              0xca0
> +#define PHB_ERR_FREEZE_ENABLE           0xca8
> +#define PHB_ERR_AIB_FENCE_ENABLE        0xcb0
> +#define PHB_ERR_LOG_0                   0xcc0
> +#define PHB_ERR_LOG_1                   0xcc8
> +#define PHB_ERR_STATUS_MASK             0xcd0
> +#define PHB_ERR1_STATUS_MASK            0xcd8
> +
> +#define PHB_OUT_ERR_STATUS              0xd00
> +#define PHB_OUT_ERR1_STATUS             0xd08
> +#define PHB_OUT_ERR_INJECT              0xd10
> +#define PHB_OUT_ERR_LEM_ENABLE          0xd18
> +#define PHB_OUT_ERR_IRQ_ENABLE          0xd20
> +#define PHB_OUT_ERR_FREEZE_ENABLE       0xd28
> +#define PHB_OUT_ERR_AIB_FENCE_ENABLE    0xd30
> +#define PHB_OUT_ERR_LOG_0               0xd40
> +#define PHB_OUT_ERR_LOG_1               0xd48
> +#define PHB_OUT_ERR_STATUS_MASK         0xd50
> +#define PHB_OUT_ERR1_STATUS_MASK        0xd58
> +
> +#define PHB_INA_ERR_STATUS              0xd80
> +#define PHB_INA_ERR1_STATUS             0xd88
> +#define PHB_INA_ERR_INJECT              0xd90
> +#define PHB_INA_ERR_LEM_ENABLE          0xd98
> +#define PHB_INA_ERR_IRQ_ENABLE          0xda0
> +#define PHB_INA_ERR_FREEZE_ENABLE       0xda8
> +#define PHB_INA_ERR_AIB_FENCE_ENABLE    0xdb0
> +#define PHB_INA_ERR_LOG_0               0xdc0
> +#define PHB_INA_ERR_LOG_1               0xdc8
> +#define PHB_INA_ERR_STATUS_MASK         0xdd0
> +#define PHB_INA_ERR1_STATUS_MASK        0xdd8
> +
> +#define PHB_INB_ERR_STATUS              0xe00
> +#define PHB_INB_ERR1_STATUS             0xe08
> +#define PHB_INB_ERR_INJECT              0xe10
> +#define PHB_INB_ERR_LEM_ENABLE          0xe18
> +#define PHB_INB_ERR_IRQ_ENABLE          0xe20
> +#define PHB_INB_ERR_FREEZE_ENABLE       0xe28
> +#define PHB_INB_ERR_AIB_FENCE_ENABLE    0xe30
> +#define PHB_INB_ERR_LOG_0               0xe40
> +#define PHB_INB_ERR_LOG_1               0xe48
> +#define PHB_INB_ERR_STATUS_MASK         0xe50
> +#define PHB_INB_ERR1_STATUS_MASK        0xe58
> +
> +/* Performance monitor & Debug registers */
> +#define PHB_TRACE_CONTROL               0xf80
> +#define PHB_PERFMON_CONFIG              0xf88
> +#define PHB_PERFMON_CTR0                0xf90
> +#define PHB_PERFMON_CTR1                0xf98
> +#define PHB_PERFMON_CTR2                0xfa0
> +#define PHB_PERFMON_CTR3                0xfa8
> +#define PHB_HOTPLUG_OVERRIDE            0xfb0
> +#define   PHB_HPOVR_FORCE_RESAMPLE      PPC_BIT(9)
> +#define   PHB_HPOVR_PRESENCE_A          PPC_BIT(10)
> +#define   PHB_HPOVR_PRESENCE_B          PPC_BIT(11)
> +#define   PHB_HPOVR_LINK_ACTIVE         PPC_BIT(12)
> +#define   PHB_HPOVR_LINK_BIFURCATED     PPC_BIT(13)
> +#define   PHB_HPOVR_LINK_LANE_SWAPPED   PPC_BIT(14)
> +
> +/*
> + * IODA2 on-chip tables
> + */
> +
> +#define IODA2_TBL_LIST          1
> +#define IODA2_TBL_LXIVT         2
> +#define IODA2_TBL_IVC_CAM       3
> +#define IODA2_TBL_RBA           4
> +#define IODA2_TBL_RCAM          5
> +#define IODA2_TBL_MRT           6
> +#define IODA2_TBL_PESTA         7
> +#define IODA2_TBL_PESTB         8
> +#define IODA2_TBL_TVT           9
> +#define IODA2_TBL_TCAM          10
> +#define IODA2_TBL_TDR           11
> +#define IODA2_TBL_M64BT         16
> +#define IODA2_TBL_M32DT         17
> +#define IODA2_TBL_PEEV          20
> +
> +/* LXIVT */
> +#define IODA2_LXIVT_SERVER              PPC_BITMASK(8, 23)
> +#define IODA2_LXIVT_PRIORITY            PPC_BITMASK(24, 31)
> +#define IODA2_LXIVT_NODE_ID             PPC_BITMASK(56, 63)
> +
> +/* IVT */
> +#define IODA2_IVT_SERVER                PPC_BITMASK(0, 23)
> +#define IODA2_IVT_PRIORITY              PPC_BITMASK(24, 31)
> +#define IODA2_IVT_GEN                   PPC_BITMASK(37, 38)
> +#define IODA2_IVT_P                     PPC_BITMASK(39, 39)
> +#define IODA2_IVT_Q                     PPC_BITMASK(47, 47)
> +#define IODA2_IVT_PE                    PPC_BITMASK(48, 63)
> +
> +/* TVT */
> +#define IODA2_TVT_TABLE_ADDR            PPC_BITMASK(0, 47)
> +#define IODA2_TVT_NUM_LEVELS            PPC_BITMASK(48, 50)
> +#define   IODA2_TVE_1_LEVEL     0
> +#define   IODA2_TVE_2_LEVELS    1
> +#define   IODA2_TVE_3_LEVELS    2
> +#define   IODA2_TVE_4_LEVELS    3
> +#define   IODA2_TVE_5_LEVELS    4
> +#define IODA2_TVT_TCE_TABLE_SIZE        PPC_BITMASK(51, 55)
> +#define IODA2_TVT_IO_PSIZE              PPC_BITMASK(59, 63)
> +
> +/* PESTA */
> +#define IODA2_PESTA_MMIO_FROZEN         PPC_BIT(0)
> +
> +/* PESTB */
> +#define IODA2_PESTB_DMA_STOPPED         PPC_BIT(0)
> +
> +/* M32DT */
> +#define IODA2_M32DT_PE_MASK             PPC_BITMASK(8, 15)
> +#define IODA2_M32DT_PE_LSH              PPC_BITLSHIFT(15)
> +
> +/* M64BT */
> +#define IODA2_M64BT_ENABLE              PPC_BIT(0)
> +#define IODA2_M64BT_SINGLE_PE           PPC_BIT(1)
> +#define IODA2_M64BT_BASE                PPC_BITMASK(2, 31)
> +#define IODA2_M64BT_MASK                PPC_BITMASK(34, 63)
> +#define IODA2_M64BT_SINGLE_BASE_MASK    PPC_BITMASK(2, 26)
> +#define IODA2_M64BT_SINGLE_BASE_LSH     PPC_BITLSHIFT(26)
> +#define IODA2_M64BT_PE_HI_MASK          PPC_BITMASK(27, 31)
> +#define IODA2_M64BT_PE_HI_LSH           PPC_BITLSHIFT(31)
> +#define IODA2_M64BT_SINGLE_MASK_MASK    PPC_BITMASK(34, 58)
> +#define IODA2_M64BT_SINGLE_MASK_LSH     PPC_BITLSHIFT(58)
> +#define IODA2_M64BT_PE_LOW_MASK         PPC_BITMASK(59, 63)
> +#define IODA2_M64BT_PE_LOW_LSH          PPC_BITLSHIFT(63)
> +
> +/*
> + * IODA2 in-memory tables
> + */
> +
> +/* PEST
> + *
> + * 2x8 bytes entries, PEST0 and PEST1
> + */
> +
> +#define IODA2_PEST0_MMIO_CAUSE          PPC_BIT(2)
> +#define IODA2_PEST0_CFG_READ            PPC_BIT(3)
> +#define IODA2_PEST0_CFG_WRITE           PPC_BIT(4)
> +#define IODA2_PEST0_TTYPE_MASK          PPC_BITMASK(5, 7)
> +#define IODA2_PEST0_TTYPE_LSH           PPC_BITLSHIFT(7)
> +#define   PEST_TTYPE_DMA_WRITE          0
> +#define   PEST_TTYPE_MSI                1
> +#define   PEST_TTYPE_DMA_READ           2
> +#define   PEST_TTYPE_DMA_READ_RESP      3
> +#define   PEST_TTYPE_MMIO_LOAD          4
> +#define   PEST_TTYPE_MMIO_STORE         5
> +#define   PEST_TTYPE_OTHER              7
> +#define IODA2_PEST0_CA_RETURN           PPC_BIT(8)
> +#define IODA2_PEST0_UTL_RTOS_TIMEOUT    PPC_BIT(8) /* Same bit as CA return */
> +#define IODA2_PEST0_UR_RETURN           PPC_BIT(9)
> +#define IODA2_PEST0_UTL_NONFATAL        PPC_BIT(10)
> +#define IODA2_PEST0_UTL_FATAL           PPC_BIT(11)
> +#define IODA2_PEST0_PARITY_UE           PPC_BIT(13)
> +#define IODA2_PEST0_UTL_CORRECTABLE     PPC_BIT(14)
> +#define IODA2_PEST0_UTL_INTERRUPT       PPC_BIT(15)
> +#define IODA2_PEST0_MMIO_XLATE          PPC_BIT(16)
> +#define IODA2_PEST0_IODA2_ERROR         PPC_BIT(16) /* Same bit as MMIO xlate */
> +#define IODA2_PEST0_TCE_PAGE_FAULT      PPC_BIT(18)
> +#define IODA2_PEST0_TCE_ACCESS_FAULT    PPC_BIT(19)
> +#define IODA2_PEST0_DMA_RESP_TIMEOUT    PPC_BIT(20)
> +#define IODA2_PEST0_AIB_SIZE_INVALID    PPC_BIT(21)
> +#define IODA2_PEST0_LEM_BIT_MASK        PPC_BITMASK(26, 31)
> +#define IODA2_PEST0_LEM_BIT_LSH         PPC_BITLSHIFT(31)
> +#define IODA2_PEST0_RID_MASK            PPC_BITMASK(32, 47)
> +#define IODA2_PEST0_RID_LSH             PPC_BITLSHIFT(47)
> +#define IODA2_PEST0_MSI_DATA_MASK       PPC_BITMASK(48, 63)
> +#define IODA2_PEST0_MSI_DATA_LSH        PPC_BITLSHIFT(63)
> +
> +#define IODA2_PEST1_FAIL_ADDR_MASK      PPC_BITMASK(3, 63)
> +#define IODA2_PEST1_FAIL_ADDR_LSH       PPC_BITLSHIFT(63)
> +
> +
> +#endif /* PCI_HOST_PNV_PHB3_REGS_H */
> diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
> index 86d5f54e5459..61af65fc0b76 100644
> --- a/include/hw/ppc/pnv.h
> +++ b/include/hw/ppc/pnv.h
> @@ -25,6 +25,7 @@
>  #include "hw/ppc/pnv_lpc.h"
>  #include "hw/ppc/pnv_psi.h"
>  #include "hw/ppc/pnv_occ.h"
> +#include "hw/pci-host/pnv_phb3.h"
>  
>  #define TYPE_PNV_CHIP "pnv-chip"
>  #define PNV_CHIP(obj) OBJECT_CHECK(PnvChip, (obj), TYPE_PNV_CHIP)
> @@ -72,6 +73,9 @@ typedef struct Pnv8Chip {
>      PnvLpcController lpc;
>      PnvPsi       psi;
>      PnvOCC       occ;
> +
> +#define PNV8_CHIP_PHB3_MAX 4
> +    PnvPHB3      phbs[PNV8_CHIP_PHB3_MAX];
>  } Pnv8Chip;
>  
>  #define TYPE_PNV9_CHIP "pnv9-chip"
> @@ -92,6 +96,7 @@ typedef struct PnvChipClass {
>      PnvChipType  chip_type;
>      uint64_t     chip_cfam_id;
>      uint64_t     cores_mask;
> +    uint32_t     num_phbs;
>  
>      hwaddr       xscom_base;
>  
> @@ -171,6 +176,23 @@ static inline bool pnv_is_power9(PnvMachineState *pnv)
>      return pnv_chip_is_power9(pnv->chips[0]);
>  }
>  
> +/*
> + * This is used by devices created on the command line to find a chip
> + * on which to attach to.
> + */
> +static inline PnvChip *pnv_get_chip(PnvMachineState *pnv, uint chip_id)
> +{
> +    int i;
> +
> +    for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
> +        if (chip->chip_id == chip_id) {
> +            return chip;
> +        }
> +    }
> +    return NULL;
> +}
> +
>  #define PNV_FDT_ADDR          0x01000000
>  #define PNV_TIMEBASE_FREQ     512000000ULL
>  
> diff --git a/include/hw/ppc/pnv_xscom.h b/include/hw/ppc/pnv_xscom.h
> index 255b26a5aaf6..e7ebffa5cf8a 100644
> --- a/include/hw/ppc/pnv_xscom.h
> +++ b/include/hw/ppc/pnv_xscom.h
> @@ -73,6 +73,15 @@ typedef struct PnvXScomInterfaceClass {
>  #define PNV_XSCOM_OCC_BASE        0x0066000
>  #define PNV_XSCOM_OCC_SIZE        0x6000
>  
> +#define PNV_XSCOM_PBCQ_NEST_BASE  0x2012000
> +#define PNV_XSCOM_PBCQ_NEST_SIZE  0x46
> +
> +#define PNV_XSCOM_PBCQ_PCI_BASE   0x9012000
> +#define PNV_XSCOM_PBCQ_PCI_SIZE   0x15
> +
> +#define PNV_XSCOM_PBCQ_SPCI_BASE  0x9013c00
> +#define PNV_XSCOM_PBCQ_SPCI_SIZE  0x5
> +
>  extern void pnv_xscom_realize(PnvChip *chip, Error **errp);
>  extern int pnv_dt_xscom(PnvChip *chip, void *fdt, int offset);
>  
> diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> index 6ac8a9392da6..966a996c2eac 100644
> --- a/include/hw/ppc/xics.h
> +++ b/include/hw/ppc/xics.h
> @@ -194,6 +194,7 @@ void icp_set_mfrr(ICPState *icp, uint8_t mfrr);
>  uint32_t icp_accept(ICPState *ss);
>  uint32_t icp_ipoll(ICPState *ss, uint32_t *mfrr);
>  void icp_eoi(ICPState *icp, uint32_t xirr);
> +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority);
>  
>  void ics_simple_write_xive(ICSState *ics, int nr, int server,
>                             uint8_t priority, uint8_t saved_priority);
> diff --git a/hw/intc/xics.c b/hw/intc/xics.c
> index b9f1a3c97214..59e2a5217dcc 100644
> --- a/hw/intc/xics.c
> +++ b/hw/intc/xics.c
> @@ -230,7 +230,7 @@ void icp_eoi(ICPState *icp, uint32_t xirr)
>      }
>  }
>  
> -static void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
> +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
>  {
>      ICPState *icp = xics_icp_get(ics->xics, server);
>  
> diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
> new file mode 100644
> index 000000000000..e11993e0e1fb
> --- /dev/null
> +++ b/hw/pci-host/pnv_phb3.c
> @@ -0,0 +1,1219 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qemu/log.h"
> +#include "qapi/visitor.h"
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "target/ppc/cpu.h"
> +#include "hw/pci-host/pnv_phb3_regs.h"
> +#include "hw/pci-host/pnv_phb3.h"
> +#include "hw/pci/pci_bridge.h"
> +#include "hw/pci/pci_bus.h"
> +#include "hw/pci/pcie_port.h"
> +#include "hw/ppc/pnv.h"
> +
> +static PCIDevice *pnv_phb3_find_cfg_dev(PnvPHB3 *phb)
> +{
> +    PCIHostState *pci = PCI_HOST_BRIDGE(phb);
> +    uint64_t addr = phb->regs[PHB_CONFIG_ADDRESS >> 3];
> +    uint8_t bus, devfn;
> +
> +    if (!(addr >> 63)) {
> +        return NULL;
> +    }
> +    bus = (addr >> 52) & 0xff;
> +    devfn = (addr >> 44) & 0xff;
> +
> +    return pci_find_device(pci->bus, bus, devfn);
> +}
> +
> +/*
> + * The CONFIG_DATA register expects little endian accesses, but as the
> + * region is big endian, we have to swap the value.
> + */
> +static void pnv_phb3_config_write(PnvPHB3 *phb, unsigned off,
> +                                  unsigned size, uint64_t val)
> +{
> +    uint32_t cfg_addr, limit;
> +    PCIDevice *pdev;
> +
> +    pdev = pnv_phb3_find_cfg_dev(phb);
> +    if (!pdev) {
> +        return;
> +    }
> +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xfff;
> +    cfg_addr |= off;
> +    limit = pci_config_size(pdev);
> +    if (limit <= cfg_addr) {
> +        /* conventional pci device can be behind pcie-to-pci bridge.
> +           256 <= addr < 4K has no effects. */
> +        return;
> +    }
> +    switch (size) {
> +    case 1:
> +        break;
> +    case 2:
> +        val = bswap16(val);
> +        break;
> +    case 4:
> +        val = bswap32(val);
> +        break;
> +    default:
> +        g_assert_not_reached();
> +    }
> +    pci_host_config_write_common(pdev, cfg_addr, limit, val, size);
> +}
> +
> +static uint64_t pnv_phb3_config_read(PnvPHB3 *phb, unsigned off,
> +                                     unsigned size)
> +{
> +    uint32_t cfg_addr, limit;
> +    PCIDevice *pdev;
> +    uint64_t val;
> +
> +    pdev = pnv_phb3_find_cfg_dev(phb);
> +    if (!pdev) {
> +        return ~0ull;
> +    }
> +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xffc;
> +    cfg_addr |= off;
> +    limit = pci_config_size(pdev);
> +    if (limit <= cfg_addr) {
> +        /* conventional pci device can be behind pcie-to-pci bridge.
> +           256 <= addr < 4K has no effects. */
> +        return ~0ull;
> +    }
> +    val = pci_host_config_read_common(pdev, cfg_addr, limit, size);
> +    switch (size) {
> +    case 1:
> +        return val;
> +    case 2:
> +        return bswap16(val);
> +    case 4:
> +        return bswap32(val);
> +    default:
> +        g_assert_not_reached();
> +    }
> +}
> +
> +static void pnv_phb3_check_m32(PnvPHB3 *phb)
> +{
> +    uint64_t base, start, size;
> +    MemoryRegion *parent;
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    if (phb->m32_mapped) {
> +        memory_region_del_subregion(phb->mr_m32.container, &phb->mr_m32);
> +        phb->m32_mapped = false;
> +    }
> +
> +    /* Disabled ? move on with life ... */
> +    if (!(phb->regs[PHB_PHB3_CONFIG >> 3] & PHB_PHB3C_M32_EN)) {
> +        return;
> +    }
> +
> +    /* Grab geometry from registers */
> +    base = phb->regs[PHB_M32_BASE_ADDR >> 3];
> +    start = phb->regs[PHB_M32_START_ADDR >> 3];
> +    size = ~(phb->regs[PHB_M32_BASE_MASK >> 3] | 0xfffc000000000000ull) + 1;
> +
> +    /* Check if it matches an enabled MMIO region in the PBCQ */
> +    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
> +        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
> +        parent = &pbcq->mmbar0;
> +        base -= pbcq->mmio0_base;
> +    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
> +        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
> +        parent = &pbcq->mmbar1;
> +        base -= pbcq->mmio1_base;
> +    } else {
> +        return;
> +    }
> +
> +    /* Create alias */
> +    memory_region_init_alias(&phb->mr_m32, OBJECT(phb), "phb3-m32",
> +                             &phb->pci_mmio, start, size);
> +    memory_region_add_subregion(parent, base, &phb->mr_m32);
> +    phb->m32_mapped = true;
> +}
> +
> +static void pnv_phb3_check_m64(PnvPHB3 *phb, uint32_t index)
> +{
> +    uint64_t base, start, size, m64;
> +    MemoryRegion *parent;
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    if (phb->m64_mapped[index]) {
> +        /* Should we destroy it in RCU friendly way... ? */
> +        memory_region_del_subregion(phb->mr_m64[index].container,
> +                                    &phb->mr_m64[index]);
> +        phb->m64_mapped[index] = false;
> +    }
> +
> +    /* Get table entry */
> +    m64 = phb->ioda_M64BT[index];
> +
> +    /* Disabled ? move on with life ... */
> +    if (!(m64 & IODA2_M64BT_ENABLE)) {
> +        return;
> +    }
> +
> +    /* Grab geometry from registers */
> +    base = GETFIELD(IODA2_M64BT_BASE, m64) << 20;
> +    if (m64 & IODA2_M64BT_SINGLE_PE) {
> +        base &= ~0x1ffffffull;
> +    }
> +    size = GETFIELD(IODA2_M64BT_MASK, m64) << 20;
> +    size |= 0xfffc000000000000ull;
> +    size = ~size + 1;
> +    start = base | (phb->regs[PHB_M64_UPPER_BITS >> 3]);
> +
> +    /* Check if it matches an enabled MMIO region in the PBCQ */
> +    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
> +        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
> +        parent = &pbcq->mmbar0;
> +        base -= pbcq->mmio0_base;
> +    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
> +        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
> +        parent = &pbcq->mmbar1;
> +        base -= pbcq->mmio1_base;
> +    } else {
> +        return;
> +    }
> +
> +    /* Create alias */
> +    memory_region_init_alias(&phb->mr_m64[index], OBJECT(phb), "phb3-m64",
> +                             &phb->pci_mmio, start, size);
> +    memory_region_add_subregion(parent, base, &phb->mr_m64[index]);
> +    phb->m64_mapped[index] = true;
> +}
> +
> +static void pnv_phb3_check_all_m64s(PnvPHB3 *phb)
> +{
> +    uint64_t i;
> +
> +    for (i = 0; i < PNV_PHB3_NUM_M64; i++) {
> +        pnv_phb3_check_m64(phb, i);
> +    }
> +}
> +
> +static void pnv_phb3_lxivt_write(PnvPHB3 *phb, unsigned idx, uint64_t val)
> +{
> +    ICSState *ics = &phb->lsis;
> +    uint8_t server, prio;
> +
> +    phb->ioda_LXIVT[idx] = val & (IODA2_LXIVT_SERVER |
> +                                  IODA2_LXIVT_PRIORITY |
> +                                  IODA2_LXIVT_NODE_ID);
> +    server = GETFIELD(IODA2_LXIVT_SERVER, val);
> +    prio = GETFIELD(IODA2_LXIVT_PRIORITY, val);
> +
> +    /*
> +     * The low order 2 bits are the link pointer (Type II interrupts).
> +     * Shift back to get a valid IRQ server.
> +     */
> +    server >>= 2;
> +
> +    ics_simple_write_xive(ics, idx, server, prio, prio);
> +}
> +
> +static uint64_t *pnv_phb3_ioda_access(PnvPHB3 *phb,
> +                                      unsigned *out_table, unsigned *out_idx)
> +{
> +    uint64_t adreg = phb->regs[PHB_IODA_ADDR >> 3];
> +    unsigned int index = GETFIELD(PHB_IODA_AD_TADR, adreg);
> +    unsigned int table = GETFIELD(PHB_IODA_AD_TSEL, adreg);
> +    unsigned int mask;
> +    uint64_t *tptr = NULL;
> +
> +    switch (table) {
> +    case IODA2_TBL_LIST:
> +        tptr = phb->ioda_LIST;
> +        mask = 7;
> +        break;
> +    case IODA2_TBL_LXIVT:
> +        tptr = phb->ioda_LXIVT;
> +        mask = 7;
> +        break;
> +    case IODA2_TBL_IVC_CAM:
> +    case IODA2_TBL_RBA:
> +        mask = 31;
> +        break;
> +    case IODA2_TBL_RCAM:
> +        mask = 63;
> +        break;
> +    case IODA2_TBL_MRT:
> +        mask = 7;
> +        break;
> +    case IODA2_TBL_PESTA:
> +    case IODA2_TBL_PESTB:
> +        mask = 255;
> +        break;
> +    case IODA2_TBL_TVT:
> +        tptr = phb->ioda_TVT;
> +        mask = 511;
> +        break;
> +    case IODA2_TBL_TCAM:
> +    case IODA2_TBL_TDR:
> +        mask = 63;
> +        break;
> +    case IODA2_TBL_M64BT:
> +        tptr = phb->ioda_M64BT;
> +        mask = 15;
> +        break;
> +    case IODA2_TBL_M32DT:
> +        tptr = phb->ioda_MDT;
> +        mask = 255;
> +        break;
> +    case IODA2_TBL_PEEV:
> +        tptr = phb->ioda_PEEV;
> +        mask = 3;
> +        break;
> +    default:
> +        return NULL;
> +    }
> +    index &= mask;
> +    if (out_idx) {
> +        *out_idx = index;
> +    }
> +    if (out_table) {
> +        *out_table = table;
> +    }
> +    if (adreg & PHB_IODA_AD_AUTOINC) {
> +        index = (index + 1) & mask;
> +        adreg = SETFIELD(PHB_IODA_AD_TADR, adreg, index);
> +    }
> +    if (tptr) {
> +        tptr += index;
> +    }
> +    phb->regs[PHB_IODA_ADDR >> 3] = adreg;
> +    return tptr;
> +}
> +
> +static uint64_t pnv_phb3_ioda_read(PnvPHB3 *phb)
> +{
> +        unsigned table;
> +        uint64_t *tptr;
> +
> +        tptr = pnv_phb3_ioda_access(phb, &table, NULL);
> +        if (!tptr) {
> +            /* Return 0 on unsupported tables, not ff's */
> +            return 0;
> +        }
> +        return *tptr;
> +}
> +
> +static void pnv_phb3_ioda_write(PnvPHB3 *phb, uint64_t val)
> +{
> +        unsigned table, idx;
> +        uint64_t *tptr;
> +
> +        tptr = pnv_phb3_ioda_access(phb, &table, &idx);
> +        if (!tptr) {
> +            return;
> +        }
> +
> +        /* Handle side effects */
> +        switch (table) {
> +        case IODA2_TBL_LXIVT:
> +            pnv_phb3_lxivt_write(phb, idx, val);
> +            break;
> +        case IODA2_TBL_M64BT:
> +            *tptr = val;
> +            pnv_phb3_check_m64(phb, idx);
> +            break;
> +        default:
> +            *tptr = val;
> +        }
> +}
> +
> +/* This is called whenever the PHB LSI, MSI source ID register or
> + * the PBCQ irq filters are written.
> + */
> +void pnv_phb3_remap_irqs(PnvPHB3 *phb)
> +{
> +    ICSState *ics = &phb->lsis;
> +    uint32_t local, global, count, mask, comp;
> +    uint64_t baren;
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    /* First check if we are enabled. Unlike real HW we don't separate TX and RX
> +     * so we enable if both are set
> +     */
> +    baren = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
> +    if (!(baren & PBCQ_NEST_BAR_EN_IRSN_RX) ||
> +        !(baren & PBCQ_NEST_BAR_EN_IRSN_TX)) {
> +        ics->offset = 0;
> +        return;
> +    }
> +
> +    /* Grab local LSI source ID */
> +    local = GETFIELD(PHB_LSI_SRC_ID, phb->regs[PHB_LSI_SOURCE_ID >> 3]) << 3;
> +
> +    /* Grab global one and compare */
> +    global = GETFIELD(PBCQ_NEST_LSI_SRC,
> +                      pbcq->nest_regs[PBCQ_NEST_LSI_SRC_ID]) << 3;
> +    if (global != local) {
> +        /* This happens during initialization, let's come back when we
> +         * are properly configured
> +         */
> +        ics->offset = 0;
> +        return;
> +    }
> +
> +    /* Get the base on the powerbus */
> +    comp = GETFIELD(PBCQ_NEST_IRSN_COMP,
> +                    pbcq->nest_regs[PBCQ_NEST_IRSN_COMPARE]);
> +    mask = GETFIELD(PBCQ_NEST_IRSN_COMP,
> +                    pbcq->nest_regs[PBCQ_NEST_IRSN_MASK]);
> +    count = ((~mask) + 1) & 0x7ffff;
> +    phb->total_irq = count;
> +
> +    /* Sanity checks */
> +    if ((global + PNV_PHB3_NUM_LSI) > count) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "LSIs out of reach: LSI base=%d total irq=%d",
> +                      global, count);
> +    }
> +
> +    if (count > 2048) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "More interrupts than supported: %d", count);
> +    }
> +
> +    if ((comp & mask) != comp) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "IRQ compare bits not in mask: comp=0x%x mask=0x%x",
> +                      comp, mask);
> +        comp &= mask;
> +    }
> +    /* Setup LSI offset */
> +    ics->offset = comp + global;
> +
> +    /* Setup MSI offset */
> +    pnv_phb3_msi_update_config(&phb->msis, comp, count - PNV_PHB3_NUM_LSI);
> +}
> +
> +static void pnv_phb3_lsi_src_id_write(PnvPHB3 *phb, uint64_t val)
> +{
> +    /* Sanitize content */
> +    val &= PHB_LSI_SRC_ID;
> +    phb->regs[PHB_LSI_SOURCE_ID >> 3] = val;
> +    pnv_phb3_remap_irqs(phb);
> +}
> +
> +static void pnv_phb3_rtc_invalidate(PnvPHB3 *phb, uint64_t val)
> +{
> +    PnvPhb3DMASpace *ds;
> +
> +    /* Always invalidate all for now ... */
> +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> +        ds->pe_num = PHB_INVALID_PE;
> +    }
> +}
> +
> +
> +static void pnv_phb3_update_msi_regions(PnvPhb3DMASpace *ds)
> +{
> +    uint64_t cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
> +
> +    if (cfg & PHB_PHB3C_32BIT_MSI_EN) {
> +        if (!ds->msi32_mapped) {
> +            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        0xffff0000, &ds->msi32_mr);
> +            ds->msi32_mapped = true;
> +        }
> +    } else {
> +        if (ds->msi32_mapped) {
> +            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        &ds->msi32_mr);
> +            ds->msi32_mapped = false;
> +        }
> +    }
> +
> +    if (cfg & PHB_PHB3C_64BIT_MSI_EN) {
> +        if (!ds->msi64_mapped) {
> +            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        (1ull << 60), &ds->msi64_mr);
> +            ds->msi64_mapped = true;
> +        }
> +    } else {
> +        if (ds->msi64_mapped) {
> +            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        &ds->msi64_mr);
> +            ds->msi64_mapped = false;
> +        }
> +    }
> +}
> +
> +static void pnv_phb3_update_all_msi_regions(PnvPHB3 *phb)
> +{
> +    PnvPhb3DMASpace *ds;
> +
> +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> +        pnv_phb3_update_msi_regions(ds);
> +    }
> +}
> +
> +void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size)
> +{
> +    PnvPHB3 *phb = opaque;
> +    bool changed;
> +
> +    /* Special case configuration data */
> +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> +        pnv_phb3_config_write(phb, off & 0x3, size, val);
> +        return;
> +    }
> +
> +    /* Other registers are 64-bit only */
> +    if (size != 8 || off & 0x7) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> +                      off, size);
> +        return;
> +    }
> +
> +    /* Handle masking */
> +    switch (off) {
> +    case PHB_M64_UPPER_BITS:
> +        val &= 0xfffc000000000000ull;
> +        break;
> +    }
> +
> +    /* Record whether it changed */
> +    changed = phb->regs[off >> 3] != val;
> +
> +    /* Store in register cache first */
> +    phb->regs[off >> 3] = val;
> +
> +    /* Handle side effects */
> +    switch (off) {
> +    case PHB_PHB3_CONFIG:
> +        if (changed) {
> +            pnv_phb3_update_all_msi_regions(phb);
> +        }
> +        /* fall through */
> +    case PHB_M32_BASE_ADDR:
> +    case PHB_M32_BASE_MASK:
> +    case PHB_M32_START_ADDR:
> +        if (changed) {
> +            pnv_phb3_check_m32(phb);
> +        }
> +        break;
> +    case PHB_M64_UPPER_BITS:
> +        if (changed) {
> +            pnv_phb3_check_all_m64s(phb);
> +        }
> +        break;
> +    case PHB_LSI_SOURCE_ID:
> +        if (changed) {
> +            pnv_phb3_lsi_src_id_write(phb, val);
> +        }
> +        break;
> +
> +    /* IODA table accesses */
> +    case PHB_IODA_DATA0:
> +        pnv_phb3_ioda_write(phb, val);
> +        break;
> +
> +    /* RTC invalidation */
> +    case PHB_RTC_INVALIDATE:
> +        pnv_phb3_rtc_invalidate(phb, val);
> +        break;
> +
> +    /* FFI request */
> +    case PHB_FFI_REQUEST:
> +        pnv_phb3_msi_ffi(&phb->msis, val);
> +        break;
> +
> +    /* Silent simple writes */
> +    case PHB_CONFIG_ADDRESS:
> +    case PHB_IODA_ADDR:
> +    case PHB_TCE_KILL:
> +    case PHB_TCE_SPEC_CTL:
> +    case PHB_PEST_BAR:
> +    case PHB_PELTV_BAR:
> +    case PHB_RTT_BAR:
> +    case PHB_RBA_BAR:
> +    case PHB_IVT_BAR:
> +    case PHB_FFI_LOCK:
> +        break;
> +
> +    /* Noise on anything else */
> +    default:
> +        qemu_log_mask(LOG_UNIMP, "reg_write 0x%"PRIx64"=%"PRIx64, off, val);
> +    }
> +}
> +
> +uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size)
> +{
> +    PnvPHB3 *phb = opaque;
> +    uint64_t val;
> +
> +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> +        return pnv_phb3_config_read(phb, off & 0x3, size);
> +    }
> +
> +    /* Other registers are 64-bit only */
> +    if (size != 8 || off & 0x7) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> +                      off, size);
> +        return ~0ull;
> +    }
> +
> +    /* Default read from cache */
> +    val = phb->regs[off >> 3];
> +
> +    switch (off) {
> +    /* Simulate venice DD2.0 */
> +    case PHB_VERSION:
> +        return 0x000000a300000005ull;
> +
> +    /* IODA table accesses */
> +    case PHB_IODA_DATA0:
> +        return pnv_phb3_ioda_read(phb);
> +
> +    /* Link training always appears trained */
> +    case PHB_PCIE_DLP_TRAIN_CTL:
> +        return PHB_PCIE_DLP_INBAND_PRESENCE | PHB_PCIE_DLP_TC_DL_LINKACT;
> +
> +    /* FFI Lock */
> +    case PHB_FFI_LOCK:
> +        /* Set lock and return previous value */
> +        phb->regs[off >> 3] |= PHB_FFI_LOCK_STATE;
> +        return val;
> +
> +    /* Silent simple reads */
> +    case PHB_PHB3_CONFIG:
> +    case PHB_M32_BASE_ADDR:
> +    case PHB_M32_BASE_MASK:
> +    case PHB_M32_START_ADDR:
> +    case PHB_CONFIG_ADDRESS:
> +    case PHB_IODA_ADDR:
> +    case PHB_RTC_INVALIDATE:
> +    case PHB_TCE_KILL:
> +    case PHB_TCE_SPEC_CTL:
> +    case PHB_PEST_BAR:
> +    case PHB_PELTV_BAR:
> +    case PHB_RTT_BAR:
> +    case PHB_RBA_BAR:
> +    case PHB_IVT_BAR:
> +    case PHB_M64_UPPER_BITS:
> +        break;
> +
> +    /* Noise on anything else */
> +    default:
> +        qemu_log_mask(LOG_UNIMP, "reg_read 0x%"PRIx64"=%"PRIx64, off, val);
> +    }
> +    return val;
> +}
> +
> +static const MemoryRegionOps pnv_phb3_reg_ops = {
> +    .read = pnv_phb3_reg_read,
> +    .write = pnv_phb3_reg_write,
> +    .valid.min_access_size = 1,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 1,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static int pnv_phb3_map_irq(PCIDevice *pci_dev, int irq_num)
> +{
> +    /* Check that out properly ... */
> +    return irq_num & 3;
> +}
> +
> +static void pnv_phb3_set_irq(void *opaque, int irq_num, int level)
> +{
> +    PnvPHB3 *phb = opaque;
> +
> +    /* LSI only ... */
> +    if (irq_num > 3) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Unknown IRQ to set %d", irq_num);
> +    }
> +    qemu_set_irq(phb->lsis.qirqs[irq_num], level);
> +}
> +
> +static bool pnv_phb3_resolve_pe(PnvPhb3DMASpace *ds)
> +{
> +    uint64_t rtt, addr;
> +    uint16_t rte;
> +    int bus_num;
> +
> +    /* Already resolved ? */
> +    if (ds->pe_num != PHB_INVALID_PE) {
> +        return true;
> +    }
> +
> +    /* We need to lookup the RTT */
> +    rtt = ds->phb->regs[PHB_RTT_BAR >> 3];
> +    if (!(rtt & PHB_RBA_BAR_ENABLE)) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "DMA with RTT BAR disabled !");
> +        /* Set error bits ? fence ? ... */
> +        return false;
> +    }
> +
> +    /* Read RTE */
> +    bus_num = pci_bus_num(ds->bus);
> +    addr = rtt & PHB_RTT_BASE_ADDRESS_MASK;
> +    addr += 2 * ((bus_num << 8) | ds->devfn);
> +    if (dma_memory_read(&address_space_memory, addr, &rte, sizeof(rte))) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read RTT entry at 0x%"PRIx64,
> +                      addr);
> +        /* Set error bits ? fence ? ... */
> +        return false;
> +    }
> +    rte = be16_to_cpu(rte);
> +
> +    /* Fail upon reading of invalid PE# */
> +    if (rte >= PNV_PHB3_NUM_PE) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "RTE for RID 0x%x invalid (%04x)",
> +                      ds->devfn, rte);
> +        /* Set error bits ? fence ? ... */
> +        return false;
> +    }
> +    ds->pe_num = rte;
> +    return true;
> +}
> +
> +static void pnv_phb3_translate_tve(PnvPhb3DMASpace *ds, hwaddr addr,
> +                                   bool is_write, uint64_t tve,
> +                                   IOMMUTLBEntry *tlb)
> +{
> +    uint64_t tta = GETFIELD(IODA2_TVT_TABLE_ADDR, tve);
> +    int32_t  lev = GETFIELD(IODA2_TVT_NUM_LEVELS, tve);
> +    uint32_t tts = GETFIELD(IODA2_TVT_TCE_TABLE_SIZE, tve);
> +    uint32_t tps = GETFIELD(IODA2_TVT_IO_PSIZE, tve);
> +
> +    /* Invalid levels */
> +    if (lev > 4) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Invalid #levels in TVE %d", lev);
> +        return;
> +    }
> +
> +    /* IO Page Size of 0 means untranslated, else use TCEs */
> +    if (tps == 0) {
> +        /* We only support non-translate in top window
> +         * XXX FIX THAT, Venice/Murano support it on bottom window
> +         * above 4G and Naples suports it on everything
> +         */
> +        if (!(tve & PPC_BIT(51))) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                          "xlate for invalid non-translate TVE");
> +            return;
> +        }
> +        /* XXX Handle boundaries */
> +
> +        /* XXX Use 4k pages like q35 ... for now */
> +        tlb->iova = addr & 0xfffffffffffff000ull;
> +        tlb->translated_addr = addr & 0x0003fffffffff000ull;
> +        tlb->addr_mask = 0xfffull;
> +        tlb->perm = IOMMU_RW;
> +    } else {
> +        uint32_t tce_shift, tbl_shift, sh;
> +        uint64_t base, taddr, tce, tce_mask;
> +
> +        /* TVE disabled ? */
> +        if (tts == 0) {
> +            qemu_log_mask(LOG_GUEST_ERROR, "xlate for invalid translated TVE");
> +            return;
> +        }
> +
> +        /* Address bits per bottom level TCE entry */
> +        tce_shift = tps + 11;
> +
> +        /* Address bits per table level */
> +        tbl_shift = tts + 8;
> +
> +        /* Top level table base address */
> +        base = tta << 12;
> +
> +        /* Total shift to first level */
> +        sh = tbl_shift * lev + tce_shift;
> +
> +        /* XXX Multi-level untested */
> +        while ((lev--) >= 0) {
> +            /* Grab the TCE address */
> +            taddr = base | (((addr >> sh) & ((1ul << tbl_shift) - 1)) << 3);
> +            if (dma_memory_read(&address_space_memory, taddr, &tce,
> +                                sizeof(tce))) {
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              "Failed to read TCE at 0x%"PRIx64, taddr);
> +                return;
> +            }
> +            tce = be64_to_cpu(tce);
> +
> +            /* Check permission for indirect TCE */
> +            if ((lev >= 0) && !(tce & 3)) {
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              "Invalid indirect TCE at 0x%"PRIx64, taddr);
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              " xlate %"PRIx64":%c TVE=%"PRIx64,
> +                              addr, is_write ? 'W' : 'R', tve);
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              " tta=%"PRIx64" lev=%d tts=%d tps=%d",
> +                              tta, lev, tts, tps);
> +                return;
> +            }
> +            sh -= tbl_shift;
> +            base = tce & ~0xfffull;
> +        }
> +
> +        /* We exit the loop with TCE being the final TCE */
> +        tce_mask = ~((1ull << tce_shift) - 1);
> +        tlb->iova = addr & tce_mask;
> +        tlb->translated_addr = tce & tce_mask;
> +        tlb->addr_mask = ~tce_mask;
> +        tlb->perm = tce & 3;
> +        if ((is_write & !(tce & 2)) || ((!is_write) && !(tce & 1))) {
> +            qemu_log_mask(LOG_GUEST_ERROR, "TCE access fault at 0x%"PRIx64,
> +                          taddr);
> +            qemu_log_mask(LOG_GUEST_ERROR, " xlate %"PRIx64":%c TVE=%"PRIx64,
> +                          addr, is_write ? 'W' : 'R', tve);
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                          " tta=%"PRIx64" lev=%d tts=%d tps=%d",
> +                          tta, lev, tts, tps);
> +        }
> +    }
> +}
> +
> +static IOMMUTLBEntry pnv_phb3_translate_iommu(IOMMUMemoryRegion *iommu,
> +                                              hwaddr addr,
> +                                              IOMMUAccessFlags flag,
> +                                              int iommu_idx)
> +{
> +    PnvPhb3DMASpace *ds = container_of(iommu, PnvPhb3DMASpace, dma_mr);
> +    int tve_sel;
> +    uint64_t tve, cfg;
> +    IOMMUTLBEntry ret = {
> +        .target_as = &address_space_memory,
> +        .iova = addr,
> +        .translated_addr = 0,
> +        .addr_mask = ~(hwaddr)0,
> +        .perm = IOMMU_NONE,
> +    };
> +
> +    /* Resolve PE# */
> +    if (!pnv_phb3_resolve_pe(ds)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> +        return ret;
> +    }
> +
> +    /* Check top bits */
> +    switch (addr >> 60) {
> +    case 00:
> +        /* DMA or 32-bit MSI ? */
> +        cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
> +        if ((cfg & PHB_PHB3C_32BIT_MSI_EN) &&
> +            ((addr & 0xffffffffffff0000ull) == 0xffff0000ull)) {
> +            qemu_log_mask(LOG_GUEST_ERROR, "xlate on 32-bit MSI region");
> +            return ret;
> +        }
> +        /* Choose TVE XXX Use PHB3 Control Register */
> +        tve_sel = (addr >> 59) & 1;
> +        tve = ds->phb->ioda_TVT[ds->pe_num * 2 + tve_sel];
> +        pnv_phb3_translate_tve(ds, addr, flag & IOMMU_WO, tve, &ret);
> +        break;
> +    case 01:
> +        qemu_log_mask(LOG_GUEST_ERROR, "xlate on 64-bit MSI region");
> +        break;
> +    default:
> +        qemu_log_mask(LOG_GUEST_ERROR, "xlate on unsupported address 0x%"PRIx64,
> +                      addr);
> +    }
> +    return ret;
> +}
> +
> +#define TYPE_PNV_PHB3_IOMMU_MEMORY_REGION "pnv-phb3-iommu-memory-region"
> +#define PNV_PHB3_IOMMU_MEMORY_REGION(obj) \
> +    OBJECT_CHECK(IOMMUMemoryRegion, (obj), TYPE_PNV_PHB3_IOMMU_MEMORY_REGION)
> +
> +static void pnv_phb3_iommu_memory_region_class_init(ObjectClass *klass,
> +                                                    void *data)
> +{
> +    IOMMUMemoryRegionClass *imrc = IOMMU_MEMORY_REGION_CLASS(klass);
> +
> +    imrc->translate = pnv_phb3_translate_iommu;
> +}
> +
> +static const TypeInfo pnv_phb3_iommu_memory_region_info = {
> +    .parent = TYPE_IOMMU_MEMORY_REGION,
> +    .name = TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
> +    .class_init = pnv_phb3_iommu_memory_region_class_init,
> +};
> +
> +/*
> + * MSI/MSIX memory region implementation.
> + * The handler handles both MSI and MSIX.
> + */
> +static void pnv_phb3_msi_write(void *opaque, hwaddr addr,
> +                               uint64_t data, unsigned size)
> +{
> +    PnvPhb3DMASpace *ds = opaque;
> +
> +    /* Resolve PE# */
> +    if (!pnv_phb3_resolve_pe(ds)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> +        return;
> +    }
> +
> +    pnv_phb3_msi_send(&ds->phb->msis, addr, data, ds->pe_num);
> +}
> +
> +static const MemoryRegionOps pnv_phb3_msi_ops = {
> +    /* There is no .read as the read result is undefined by PCI spec */
> +    .read = NULL,
> +    .write = pnv_phb3_msi_write,
> +    .endianness = DEVICE_LITTLE_ENDIAN
> +};
> +
> +static AddressSpace *pnv_phb3_dma_iommu(PCIBus *bus, void *opaque, int devfn)
> +{
> +    PnvPHB3 *phb = opaque;
> +    PnvPhb3DMASpace *ds;
> +
> +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> +        if (ds->bus == bus && ds->devfn == devfn) {
> +            break;
> +        }
> +    }
> +
> +    if (ds == NULL) {
> +        ds = g_malloc0(sizeof(PnvPhb3DMASpace));
> +        ds->bus = bus;
> +        ds->devfn = devfn;
> +        ds->pe_num = PHB_INVALID_PE;
> +        ds->phb = phb;
> +        memory_region_init_iommu(&ds->dma_mr, sizeof(ds->dma_mr),
> +                                 TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
> +                                 OBJECT(phb), "phb3_iommu", UINT64_MAX);
> +        address_space_init(&ds->dma_as, MEMORY_REGION(&ds->dma_mr),
> +                           "phb3_iommu");
> +        memory_region_init_io(&ds->msi32_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> +                              ds, "msi32", 0x10000);
> +        memory_region_init_io(&ds->msi64_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> +                              ds, "msi64", 0x100000);
> +        pnv_phb3_update_msi_regions(ds);
> +
> +        QLIST_INSERT_HEAD(&phb->dma_spaces, ds, list);
> +    }
> +    return &ds->dma_as;
> +}
> +
> +static void pnv_phb3_instance_init(Object *obj)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(obj);
> +
> +    /* Create LSI source */
> +    object_initialize(&phb->lsis, sizeof(phb->lsis), TYPE_ICS_SIMPLE);
> +    object_property_add_child(obj, "ics-phb-lsi", OBJECT(&phb->lsis), NULL);
> +
> +    /* Default init ... will be fixed by HW inits */
> +    phb->lsis.offset = 0;
> +
> +    /* Create MSI source */
> +    object_initialize(&phb->msis, sizeof(phb->msis), TYPE_PHB3_MSI);
> +    object_property_add_const_link(OBJECT(&phb->msis), "phb", obj,
> +                                   &error_abort);
> +    object_property_add_child(obj, "ics-phb-msi", OBJECT(&phb->msis), NULL);
> +
> +    /* Create PBCQ */
> +    object_initialize(&phb->pbcq, sizeof(phb->pbcq), TYPE_PNV_PBCQ);
> +    object_property_add_const_link(OBJECT(&phb->pbcq), "phb", obj,
> +                                   &error_abort);
> +    object_property_add_child(obj, "pbcq", OBJECT(&phb->pbcq), NULL);
> +
> +    QLIST_INIT(&phb->dma_spaces);
> +}
> +
> +/*
> + * This could be done under pnv_pbcq_realize
> + */
> +static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
> +{
> +    PCIHostState *pcih = PCI_HOST_BRIDGE(phb);
> +    PCIDevice *brdev;
> +    PCIDevice *pdev;
> +    PCIBus *parent;
> +    uint8_t chassis = phb->chip_id * 4 + phb->phb_id;
> +    uint8_t chassis_nr = 128;
> +
> +    /* Add root complex */
> +    pdev = pci_create(pcih->bus, 0, TYPE_PNV_PHB3_RC);
> +    object_property_add_child(OBJECT(phb), "phb3-rc", OBJECT(pdev), NULL);
> +    qdev_prop_set_uint8(DEVICE(pdev), "chassis", chassis);
> +    qdev_prop_set_uint16(DEVICE(pdev), "slot", 1);
> +    qdev_init_nofail(DEVICE(pdev));
> +
> +    /* Setup bus for that chip */
> +    parent = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
> +
> +    brdev = pci_create(parent, 0, "pci-bridge");
> +    object_property_add_child(OBJECT(parent), "pci-bridge", OBJECT(brdev),
> +                              NULL);
> +    qdev_prop_set_uint8(DEVICE(brdev), PCI_BRIDGE_DEV_PROP_CHASSIS_NR,
> +                        chassis_nr);
> +    qdev_init_nofail(DEVICE(brdev));
> +}
> +
> +static void pnv_phb3_realize(DeviceState *dev, Error **errp)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(dev);
> +    PCIHostState *pci = PCI_HOST_BRIDGE(dev);
> +    Object *xics = OBJECT(qdev_get_machine());
> +    Error *local_err = NULL;
> +    int i;
> +
> +    memory_region_init(&phb->pci_mmio, OBJECT(phb), "pci-mmio",
> +                       PCI_MMIO_TOTAL_SIZE);
> +
> +    /* PHB3 doesn't support IO space. However, qemu gets very upset if
> +     * we don't have an IO region to anchor IO BARs onto so we just
> +     * initialize one which we never hook up to anything
> +     */
> +    memory_region_init(&phb->pci_io, OBJECT(phb), "pci-io", 0x10000);
> +
> +    memory_region_init_io(&phb->mr_regs, OBJECT(phb), &pnv_phb3_reg_ops, phb,
> +                          "phb3-regs", 0x1000);
> +
> +    object_property_set_int(OBJECT(&phb->lsis), PNV_PHB3_NUM_LSI, "nr-irqs",
> +                            &error_abort);
> +    object_property_add_const_link(OBJECT(&phb->lsis), "xics", xics,
> +                                   &error_abort);
> +    object_property_set_bool(OBJECT(&phb->lsis), true, "realized", &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    for (i = 0; i < PNV_PHB3_NUM_LSI; i++) {
> +        ics_set_irq_type(&phb->lsis, i, true);
> +    }
> +
> +    object_property_add_const_link(OBJECT(&phb->msis), "xics", xics,
> +                                   &error_abort);
> +    object_property_set_int(OBJECT(&phb->msis), PHB3_MAX_MSI, "nr-irqs",
> +                            &error_abort);
> +    object_property_set_bool(OBJECT(&phb->msis), true, "realized", &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    object_property_set_bool(OBJECT(&phb->pbcq), true, "realized", &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    pci->bus = pci_register_root_bus(dev, "phb3-root-bus",
> +                                pnv_phb3_set_irq, pnv_phb3_map_irq, phb,
> +                                &phb->pci_mmio, &phb->pci_io,
> +                                0, 4, TYPE_PNV_PHB3_ROOT_BUS);
> +    pci_setup_iommu(pci->bus, pnv_phb3_dma_iommu, phb);
> +
> +    /* Setup the PCI busses */
> +    pnv_phb3_pci_create(phb, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +}
> +
> +void pnv_phb3_update_regions(PnvPHB3 *phb)
> +{
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    /* Unmap first always */
> +    if (phb->regs_mapped) {
> +        memory_region_del_subregion(&pbcq->phbbar, &phb->mr_regs);
> +        phb->regs_mapped = false;
> +    }
> +
> +    /* Map registers if enabled */
> +    if (pbcq->phb_mapped) {
> +        /* XXX We should use the PHB BAR 2 register but we don't ... */
> +        memory_region_add_subregion(&pbcq->phbbar, 0, &phb->mr_regs);
> +        phb->regs_mapped = true;
> +    }
> +
> +    /* Check/update m32 */
> +    if (phb->m32_mapped) {
> +        pnv_phb3_check_m32(phb);
> +    }
> +}
> +
> +static const char *pnv_phb3_root_bus_path(PCIHostState *host_bridge,
> +                                          PCIBus *rootbus)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(host_bridge);
> +
> +    snprintf(phb->bus_path, sizeof(phb->bus_path), "00%02x:%02x",
> +             phb->chip_id, phb->phb_id);
> +    return phb->bus_path;
> +}
> +
> +static void pnv_phb3_get_phb_id(Object *obj, Visitor *v, const char *name,
> +                              void *opaque, Error **errp)
> +{
> +    Property *prop = opaque;
> +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> +
> +    visit_type_uint32(v, name, ptr, errp);
> +}
> +
> +static void pnv_phb3_set_phb_id(Object *obj, Visitor *v, const char *name,
> +                              void *opaque, Error **errp)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(obj);
> +    uint32_t phb_id;
> +    Error *local_err = NULL;
> +
> +    visit_type_uint32(v, name, &phb_id, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    /*
> +     * Limit to a maximum of 6 PHBs per chip
> +     */
> +    if (phb_id >= PNV8_CHIP_PHB3_MAX) {
> +        error_setg(errp, "invalid PHB index: '%d'", phb_id);
> +        return;
> +    }
> +
> +    phb->phb_id = phb_id;
> +}
> +
> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
> +    .name = "irq",
> +    .get = pnv_phb3_get_phb_id,
> +    .set = pnv_phb3_set_phb_id,
> +};
> +
> +static void pnv_phb3_get_chip_id(Object *obj, Visitor *v, const char *name,
> +                              void *opaque, Error **errp)
> +{
> +    Property *prop = opaque;
> +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> +
> +    visit_type_uint32(v, name, ptr, errp);
> +}
> +
> +static void pnv_phb3_set_chip_id(Object *obj, Visitor *v, const char *name,
> +                                 void *opaque, Error **errp)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(obj);
> +    uint32_t chip_id;
> +    Error *local_err = NULL;
> +    PnvChip *chip;
> +
> +    visit_type_uint32(v, name, &chip_id, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    /*
> +     * Check that the chip owning the device is valid.
> +     */
> +    chip = pnv_get_chip(PNV_MACHINE(qdev_get_machine()), chip_id);
> +    if (!chip) {
> +        error_setg(errp, "chip id '%d' is invalid", chip_id);
> +        return;
> +    }
> +
> +    phb->chip_id = chip_id;
> +}
> +
> +static const PropertyInfo pnv_phb3_chip_id_propinfo = {
> +    .name = "irq",
> +    .get = pnv_phb3_get_chip_id,
> +    .set = pnv_phb3_set_chip_id,
> +};
> +
> +static Property pnv_phb3_properties[] = {
> +    DEFINE_PROP("phb-id", PnvPHB3, phb_id, pnv_phb3_phb_id_propinfo,
> +                uint32_t),
> +    DEFINE_PROP("chip-id", PnvPHB3, chip_id, pnv_phb3_chip_id_propinfo,
> +                uint32_t),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static void pnv_phb3_class_init(ObjectClass *klass, void *data)
> +{
> +    PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_CLASS(klass);
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +
> +    hc->root_bus_path = pnv_phb3_root_bus_path;
> +    dc->realize = pnv_phb3_realize;
> +    dc->props = pnv_phb3_properties;
> +    set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
> +}
> +
> +static const TypeInfo pnv_phb3_type_info = {
> +    .name          = TYPE_PNV_PHB3,
> +    .parent        = TYPE_PCI_HOST_BRIDGE,
> +    .instance_size = sizeof(PnvPHB3),
> +    .class_init    = pnv_phb3_class_init,
> +    .instance_init = pnv_phb3_instance_init,
> +};
> +
> +static void pnv_phb3_root_bus_class_init(ObjectClass *klass, void *data)
> +{
> +    BusClass *k = BUS_CLASS(klass);
> +
> +    k->max_dev = 1;
> +}
> +
> +static const TypeInfo pnv_phb3_root_bus_info = {
> +    .name = TYPE_PNV_PHB3_ROOT_BUS,
> +    .parent = TYPE_PCIE_BUS,
> +    .class_init = pnv_phb3_root_bus_class_init,
> +};
> +
> +static void pnv_phb3_rc_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
> +    PCIERootPortClass *rpc = PCIE_ROOT_PORT_CLASS(klass);
> +
> +    dc->desc     = "IBM PHB3 PCIE Root Port";
> +
> +    k->vendor_id = PCI_VENDOR_ID_IBM;
> +    k->device_id = 0x03dc;
> +    k->revision  = 0;
> +    rpc->exp_offset = 0x48;
> +    rpc->aer_offset = 0x100;
> +}
> +
> +static const TypeInfo pnv_phb3_rc_info = {
> +    .name          = TYPE_PNV_PHB3_RC,
> +    .parent        = TYPE_PCIE_ROOT_PORT,
> +    .class_init    = pnv_phb3_rc_class_init,
> +};
> +
> +static void pnv_phb3_register_types(void)
> +{
> +    type_register_static(&pnv_phb3_rc_info);
> +    type_register_static(&pnv_phb3_type_info);
> +    type_register_static(&pnv_phb3_root_bus_info);
> +    type_register_static(&pnv_phb3_iommu_memory_region_info);
> +}
> +
> +type_init(pnv_phb3_register_types)
> diff --git a/hw/pci-host/pnv_phb3_msi.c b/hw/pci-host/pnv_phb3_msi.c
> new file mode 100644
> index 000000000000..36927da25b61
> --- /dev/null
> +++ b/hw/pci-host/pnv_phb3_msi.c
> @@ -0,0 +1,316 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qemu/log.h"
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "target/ppc/cpu.h"
> +#include "hw/pci-host/pnv_phb3_regs.h"
> +#include "hw/pci-host/pnv_phb3.h"
> +#include "hw/ppc/pnv.h"
> +#include "hw/pci/msi.h"
> +
> +static uint64_t phb3_msi_ive_addr(PnvPHB3 *phb, int srcno)
> +{
> +    uint64_t ivtbar = phb->regs[PHB_IVT_BAR >> 3];
> +    uint64_t phbctl = phb->regs[PHB_CONTROL >> 3];
> +
> +    if (!(ivtbar & PHB_IVT_BAR_ENABLE)) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Failed access to disable IVT BAR !");
> +        return 0;
> +    }
> +
> +    if (srcno >= (ivtbar & PHB_IVT_LENGTH_MASK)) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "MSI out of bounds (%d vs  0x%"PRIx64")",
> +                      srcno, ivtbar & PHB_IVT_LENGTH_MASK);
> +        return 0;
> +    }
> +
> +    ivtbar &= PHB_IVT_BASE_ADDRESS_MASK;
> +
> +    if (phbctl & PHB_CTRL_IVE_128_BYTES) {
> +        return ivtbar + 128 * srcno;
> +    } else {
> +        return ivtbar + 16 * srcno;
> +    }
> +}
> +
> +static bool phb3_msi_read_ive(PnvPHB3 *phb, int srcno, uint64_t *out_ive)
> +{
> +    uint64_t ive_addr, ive;
> +
> +    ive_addr = phb3_msi_ive_addr(phb, srcno);
> +    if (!ive_addr) {
> +        return false;
> +    }
> +
> +    if (dma_memory_read(&address_space_memory, ive_addr, &ive, sizeof(ive))) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read IVE at 0x%" PRIx64,
> +                      ive_addr);
> +        return false;
> +    }
> +    *out_ive = be64_to_cpu(ive);
> +
> +    return true;
> +}
> +
> +static void phb3_msi_set_p(Phb3MsiState *msi, int srcno, uint8_t gen)
> +{
> +    uint64_t ive_addr;
> +    uint8_t p = 0x01 | (gen << 1);
> +
> +    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
> +    if (!ive_addr) {
> +        return;
> +    }
> +
> +    if (dma_memory_write(&address_space_memory, ive_addr + 4, &p, 1)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to write IVE (set P) at 0x%" PRIx64, ive_addr);
> +    }
> +}
> +
> +static void phb3_msi_set_q(Phb3MsiState *msi, int srcno)
> +{
> +    uint64_t ive_addr;
> +    uint8_t q = 0x01;
> +
> +    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
> +    if (!ive_addr) {
> +        return;
> +    }
> +
> +    if (dma_memory_write(&address_space_memory, ive_addr + 5, &q, 1)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to write IVE (set Q) at 0x%" PRIx64, ive_addr);
> +    }
> +}
> +
> +static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
> +{
> +    ICSState *ics = ICS_BASE(msi);
> +    uint64_t ive;
> +    uint64_t server, prio, pq, gen;
> +
> +    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
> +        return;
> +    }
> +
> +    server = GETFIELD(IODA2_IVT_SERVER, ive);
> +    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
> +    pq = GETFIELD(IODA2_IVT_Q, ive);
> +    if (!ignore_p) {
> +        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
> +    }
> +    gen = GETFIELD(IODA2_IVT_GEN, ive);
> +
> +    /*
> +     * The low order 2 bits are the link pointer (Type II interrupts).
> +     * Shift back to get a valid IRQ server.
> +     */
> +    server >>= 2;
> +
> +    switch (pq) {
> +    case 0: /* 00 */
> +        if (prio == 0xff) {
> +            /* Masked, set Q */
> +            phb3_msi_set_q(msi, srcno);
> +        } else {
> +            /* Enabled, set P and send */
> +            phb3_msi_set_p(msi, srcno, gen);
> +            icp_irq(ics, server, srcno + ics->offset, prio);
> +        }
> +        break;
> +    case 2: /* 10 */
> +        /* Already pending, set Q */
> +        phb3_msi_set_q(msi, srcno);
> +        break;
> +    case 1: /* 01 */
> +    case 3: /* 11 */
> +    default:
> +        /* Just drop stuff if Q already set */
> +        break;
> +    }
> +}
> +
> +static void phb3_msi_set_irq(void *opaque, int srcno, int val)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(opaque);
> +
> +    if (val) {
> +        phb3_msi_try_send(msi, srcno, false);
> +    }
> +}
> +
> +
> +void pnv_phb3_msi_send(Phb3MsiState *msi, uint64_t addr, uint16_t data,
> +                       int32_t dev_pe)
> +{
> +    ICSState *ics = ICS_BASE(msi);
> +    uint64_t ive;
> +    uint16_t pe;
> +    uint32_t src = ((addr >> 4) & 0xffff) | (data & 0x1f);
> +
> +    if (src >= ics->nr_irqs) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "MSI %d out of bounds", src);
> +        return;
> +    }
> +    if (dev_pe >= 0) {
> +        if (!phb3_msi_read_ive(msi->phb, src, &ive)) {
> +            return;
> +        }
> +        pe = GETFIELD(IODA2_IVT_PE, ive);
> +        if (pe != dev_pe) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                          "MSI %d send by PE#%d but assigned to PE#%d",
> +                          src, dev_pe, pe);
> +            return;
> +        }
> +    }
> +    qemu_irq_pulse(ics->qirqs[src]);
> +}
> +
> +void pnv_phb3_msi_ffi(Phb3MsiState *msi, uint64_t val)
> +{
> +    /* Emit interrupt */
> +    pnv_phb3_msi_send(msi, val, 0, -1);
> +
> +    /* Clear FFI lock */
> +    msi->phb->regs[PHB_FFI_LOCK >> 3] = 0;
> +}
> +
> +static void phb3_msi_reject(ICSState *ics, uint32_t nr)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(ics);
> +    unsigned int srcno = nr - ics->offset;
> +    unsigned int idx = srcno >> 6;
> +    unsigned int bit = 1ull << (srcno & 0x3f);
> +
> +    assert(srcno < PHB3_MAX_MSI);
> +
> +    msi->rba[idx] |= bit;
> +    msi->rba_sum |= (1u << idx);
> +}
> +
> +static void phb3_msi_resend(ICSState *ics)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(ics);
> +    unsigned int i, j;
> +
> +    if (msi->rba_sum == 0) {
> +        return;
> +    }
> +
> +    for (i = 0; i < 32; i++) {
> +        if ((msi->rba_sum & (1u << i)) == 0) {
> +            continue;
> +        }
> +        msi->rba_sum &= ~(1u << i);
> +        for (j = 0; j < 64; j++) {
> +            if ((msi->rba[i] & (1ull << j)) == 0) {
> +                continue;
> +            }
> +            msi->rba[i] &= ~(1u << j);
> +            phb3_msi_try_send(msi, i * 64 + j, true);
> +        }
> +    }
> +}
> +
> +static void phb3_msi_reset(DeviceState *dev)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(dev);
> +    ICSStateClass *icsc = ICS_BASE_GET_CLASS(dev);
> +
> +    icsc->parent_reset(dev);
> +
> +    memset(msi->rba, 0, sizeof(msi->rba));
> +    msi->rba_sum = 0;
> +}
> +
> +static void phb3_msi_reset_handler(void *dev)
> +{
> +    phb3_msi_reset(dev);
> +}
> +
> +void pnv_phb3_msi_update_config(Phb3MsiState *msi, uint32_t base,
> +                                uint32_t count)
> +{
> +    ICSState *ics = ICS_BASE(msi);
> +
> +    if (count > PHB3_MAX_MSI) {
> +        count = PHB3_MAX_MSI;
> +    }
> +    ics->nr_irqs = count;
> +    ics->offset = base;
> +}
> +
> +static void phb3_msi_realize(DeviceState *dev, Error **errp)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(dev);
> +    ICSState *ics = ICS_BASE(msi);
> +    ICSStateClass *icsc = ICS_BASE_GET_CLASS(ics);
> +    Object *obj;
> +    Error *local_err = NULL;
> +
> +    icsc->parent_realize(dev, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
> +    if (!obj) {
> +        error_propagate(errp, local_err);
> +        error_prepend(errp, "required link 'phb' not found: ");
> +        return;
> +    }
> +    msi->phb = PNV_PHB3(obj);
> +
> +    ics->qirqs = qemu_allocate_irqs(phb3_msi_set_irq, msi, PHB3_MAX_MSI);
> +
> +    qemu_register_reset(phb3_msi_reset_handler, dev);
> +}
> +
> +static void phb3_msi_instance_init(Object *obj)
> +{
> +    ICSState *ics = ICS_BASE(obj);
> +
> +    /* Will be overriden later */
> +    ics->offset = 0;
> +}
> +
> +static void phb3_msi_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    ICSStateClass *isc = ICS_BASE_CLASS(klass);
> +
> +    device_class_set_parent_realize(dc, phb3_msi_realize,
> +                                    &isc->parent_realize);
> +    device_class_set_parent_reset(dc, phb3_msi_reset,
> +                                  &isc->parent_reset);
> +
> +    isc->reject = phb3_msi_reject;
> +    isc->resend = phb3_msi_resend;
> +}
> +
> +static const TypeInfo phb3_msi_info = {
> +    .name = TYPE_PHB3_MSI,
> +    .parent = TYPE_ICS_BASE,
> +    .instance_size = sizeof(Phb3MsiState),
> +    .class_init = phb3_msi_class_init,
> +    .class_size = sizeof(ICSStateClass),
> +    .instance_init = phb3_msi_instance_init,
> +};
> +
> +static void pnv_phb3_msi_register_types(void)
> +{
> +    type_register_static(&phb3_msi_info);
> +}
> +
> +type_init(pnv_phb3_msi_register_types)
> diff --git a/hw/pci-host/pnv_phb3_pbcq.c b/hw/pci-host/pnv_phb3_pbcq.c
> new file mode 100644
> index 000000000000..34df3ac684e4
> --- /dev/null
> +++ b/hw/pci-host/pnv_phb3_pbcq.c
> @@ -0,0 +1,347 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "qemu/log.h"
> +#include "target/ppc/cpu.h"
> +#include "hw/ppc/fdt.h"
> +#include "hw/pci-host/pnv_phb3_regs.h"
> +#include "hw/pci-host/pnv_phb3.h"
> +#include "hw/ppc/pnv_xscom.h"
> +#include "hw/pci/pci_bridge.h"
> +#include "hw/pci/pci_bus.h"
> +
> +#include <libfdt.h>
> +
> +static uint64_t pnv_pbcq_nest_xscom_read(void *opaque, hwaddr addr,
> +                                         unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t offset = addr >> 3;
> +
> +    return pbcq->nest_regs[offset];
> +}
> +
> +static uint64_t pnv_pbcq_pci_xscom_read(void *opaque, hwaddr addr,
> +                                        unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t offset = addr >> 3;
> +
> +    return pbcq->pci_regs[offset];
> +}
> +
> +static uint64_t pnv_pbcq_spci_xscom_read(void *opaque, hwaddr addr,
> +                                         unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t offset = addr >> 3;
> +
> +    if (offset == PBCQ_SPCI_ASB_DATA) {
> +        return pnv_phb3_reg_read(pbcq->phb,
> +                                 pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR], 8);
> +    }
> +    return pbcq->spci_regs[offset];
> +}
> +
> +static void pnv_pbcq_update_map(PnvPBCQState *pbcq)
> +{
> +    uint64_t bar_en = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
> +    uint64_t bar, mask, size;
> +
> +    /*
> +     * NOTE: This will really not work well if those are remapped
> +     * after the PHB has created its sub regions. We could do better
> +     * if we had a way to resize regions but we don't really care
> +     * that much in practice as the stuff below really only happens
> +     * once early during boot
> +     */
> +
> +    /* Handle unmaps */
> +    if (pbcq->mmio0_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
> +        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar0);
> +        pbcq->mmio0_mapped = false;
> +    }
> +    if (pbcq->mmio1_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
> +        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar1);
> +        pbcq->mmio1_mapped = false;
> +    }
> +    if (pbcq->phb_mapped && !(bar_en & PBCQ_NEST_BAR_EN_PHB)) {
> +        memory_region_del_subregion(get_system_memory(), &pbcq->phbbar);
> +        pbcq->phb_mapped = false;
> +    }
> +
> +    /* Update PHB */
> +    pnv_phb3_update_regions(pbcq->phb);
> +
> +    /* Handle maps */
> +    if (!pbcq->mmio0_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
> +        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] >> 14;
> +        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0];
> +        size = ((~mask) >> 14) + 1;
> +        memory_region_init(&pbcq->mmbar0, OBJECT(pbcq), "pbcq-mmio0", size);
> +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar0);
> +        pbcq->mmio0_mapped = true;
> +        pbcq->mmio0_base = bar;
> +        pbcq->mmio0_size = size;
> +    }
> +    if (!pbcq->mmio1_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
> +        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] >> 14;
> +        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1];
> +        size = ((~mask) >> 14) + 1;
> +        memory_region_init(&pbcq->mmbar1, OBJECT(pbcq), "pbcq-mmio1", size);
> +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar1);
> +        pbcq->mmio1_mapped = true;
> +        pbcq->mmio1_base = bar;
> +        pbcq->mmio1_size = size;
> +    }
> +    if (!pbcq->phb_mapped && (bar_en & PBCQ_NEST_BAR_EN_PHB)) {
> +        bar = pbcq->nest_regs[PBCQ_NEST_PHB_BAR] >> 14;
> +        size = 0x1000;
> +        memory_region_init(&pbcq->phbbar, OBJECT(pbcq), "pbcq-phb", size);
> +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->phbbar);
> +        pbcq->phb_mapped = true;
> +    }
> +
> +    /* Update PHB */
> +    pnv_phb3_update_regions(pbcq->phb);
> +}
> +
> +static void pnv_pbcq_nest_xscom_write(void *opaque, hwaddr addr,
> +                                uint64_t val, unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t reg = addr >> 3;
> +
> +    switch (reg) {
> +    case PBCQ_NEST_MMIO_BAR0:
> +    case PBCQ_NEST_MMIO_BAR1:
> +    case PBCQ_NEST_MMIO_MASK0:
> +    case PBCQ_NEST_MMIO_MASK1:
> +        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] &
> +            (PBCQ_NEST_BAR_EN_MMIO0 |
> +             PBCQ_NEST_BAR_EN_MMIO1)) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                           "PHB3: Changing enabled BAR unsupported\n");
> +        }
> +        pbcq->nest_regs[reg] = val & 0xffffffffc0000000ull;
> +        break;
> +    case PBCQ_NEST_PHB_BAR:
> +        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] & PBCQ_NEST_BAR_EN_PHB) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                           "PHB3: Changing enabled BAR unsupported\n");
> +        }
> +        pbcq->nest_regs[reg] = val & 0xfffffffffc000000ull;
> +        break;
> +    case PBCQ_NEST_BAR_EN:
> +        pbcq->nest_regs[reg] = val & 0xf800000000000000ull;
> +        pnv_pbcq_update_map(pbcq);
> +        pnv_phb3_remap_irqs(pbcq->phb);
> +        break;
> +    case PBCQ_NEST_IRSN_COMPARE:
> +    case PBCQ_NEST_IRSN_MASK:
> +        pbcq->nest_regs[reg] = val & PBCQ_NEST_IRSN_COMP;
> +        pnv_phb3_remap_irqs(pbcq->phb);
> +        break;
> +    case PBCQ_NEST_LSI_SRC_ID:
> +        pbcq->nest_regs[reg] = val & PBCQ_NEST_LSI_SRC;
> +        pnv_phb3_remap_irqs(pbcq->phb);
> +        break;
> +    }
> +
> +    /* XXX Don't error out on other regs for now ... */
> +}
> +
> +static void pnv_pbcq_pci_xscom_write(void *opaque, hwaddr addr,
> +                                     uint64_t val, unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t reg = addr >> 3;
> +
> +    switch (reg) {
> +    case PBCQ_PCI_BAR2:
> +        pbcq->pci_regs[reg] = val & 0xfffffffffc000000ull;
> +        pnv_pbcq_update_map(pbcq);
> +        break;
> +    }
> +
> +    /* XXX Don't error out on other regs for now ... */
> +}
> +
> +static void pnv_pbcq_spci_xscom_write(void *opaque, hwaddr addr,
> +                                uint64_t val, unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t reg = addr >> 3;
> +
> +    switch (reg) {
> +    case PBCQ_SPCI_ASB_ADDR:
> +        pbcq->spci_regs[reg] = val & 0xfff;
> +        break;
> +    case PBCQ_SPCI_ASB_STATUS:
> +        pbcq->spci_regs[reg] &= ~val;
> +        break;
> +    case PBCQ_SPCI_ASB_DATA:
> +        pnv_phb3_reg_write(pbcq->phb, pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR],
> +                           val, 8);
> +        break;
> +    case PBCQ_SPCI_AIB_CAPP_EN:
> +    case PBCQ_SPCI_CAPP_SEC_TMR:
> +        break;
> +    }
> +
> +    /* XXX Don't error out on other regs for now ... */
> +}
> +
> +static const MemoryRegionOps pnv_pbcq_nest_xscom_ops = {
> +    .read = pnv_pbcq_nest_xscom_read,
> +    .write = pnv_pbcq_nest_xscom_write,
> +    .valid.min_access_size = 8,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 8,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static const MemoryRegionOps pnv_pbcq_pci_xscom_ops = {
> +    .read = pnv_pbcq_pci_xscom_read,
> +    .write = pnv_pbcq_pci_xscom_write,
> +    .valid.min_access_size = 8,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 8,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static const MemoryRegionOps pnv_pbcq_spci_xscom_ops = {
> +    .read = pnv_pbcq_spci_xscom_read,
> +    .write = pnv_pbcq_spci_xscom_write,
> +    .valid.min_access_size = 8,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 8,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static void pnv_pbcq_default_bars(PnvPBCQState *pbcq)
> +{
> +    uint64_t mm0, mm1, reg;
> +    PnvPHB3 *phb = pbcq->phb;
> +
> +    mm0 = 0x3d00000000000ull + 0x4000000000ull * phb->chip_id +
> +            0x1000000000ull * phb->phb_id;
> +    mm1 = 0x3ff8000000000ull + 0x0200000000ull * phb->chip_id +
> +            0x0080000000ull * phb->phb_id;
> +    reg = 0x3fffe40000000ull + 0x0000400000ull * phb->chip_id +
> +            0x0000100000ull * phb->phb_id;
> +
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] = mm0 << 14;
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] = mm1 << 14;
> +    pbcq->nest_regs[PBCQ_NEST_PHB_BAR] = reg << 14;
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0] = 0x3fff000000000ull << 14;
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1] = 0x3ffff80000000ull << 14;
> +    pbcq->pci_regs[PBCQ_PCI_BAR2] = reg << 14;
> +}
> +
> +static void pnv_pbcq_realize(DeviceState *dev, Error **errp)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(dev);
> +    PnvPHB3 *phb;
> +    Object *obj;
> +    Error *local_err = NULL;
> +    char name[32];
> +
> +    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
> +    if (!obj) {
> +        error_propagate(errp, local_err);
> +        error_prepend(errp, "required link 'phb' not found: ");
> +        return;
> +    }
> +    phb = pbcq->phb = PNV_PHB3(obj);
> +
> +    /* XXX Fix OPAL to do that: establish default BAR values */
> +    pnv_pbcq_default_bars(pbcq);
> +
> +    /* Initialize the XSCOM region for the PBCQ registers */
> +    snprintf(name, sizeof(name), "xscom-pbcq-nest-%d.%d",
> +             phb->chip_id, phb->phb_id);
> +    pnv_xscom_region_init(&pbcq->xscom_nest_regs, OBJECT(dev),
> +                          &pnv_pbcq_nest_xscom_ops, pbcq, name,
> +                          PNV_XSCOM_PBCQ_NEST_SIZE);
> +    snprintf(name, sizeof(name), "xscom-pbcq-pci-%d.%d",
> +             phb->chip_id, phb->phb_id);
> +    pnv_xscom_region_init(&pbcq->xscom_pci_regs, OBJECT(dev),
> +                          &pnv_pbcq_pci_xscom_ops, pbcq, name,
> +                          PNV_XSCOM_PBCQ_PCI_SIZE);
> +    snprintf(name, sizeof(name), "xscom-pbcq-spci-%d.%d",
> +             phb->chip_id, phb->phb_id);
> +    pnv_xscom_region_init(&pbcq->xscom_spci_regs, OBJECT(dev),
> +                          &pnv_pbcq_spci_xscom_ops, pbcq, name,
> +                          PNV_XSCOM_PBCQ_SPCI_SIZE);
> +}
> +
> +static int pnv_pbcq_dt_xscom(PnvXScomInterface *dev, void *fdt,
> +                             int xscom_offset)
> +{
> +    const char compat[] = "ibm,power8-pbcq";
> +    PnvPHB3 *phb = PNV_PBCQ(dev)->phb;
> +    char *name;
> +    int offset;
> +    uint32_t lpc_pcba = PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id;
> +    uint32_t reg[] = {
> +        cpu_to_be32(lpc_pcba),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_NEST_SIZE),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_SIZE),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_SIZE)
> +    };
> +
> +    name = g_strdup_printf("pbcq@%x", lpc_pcba);
> +    offset = fdt_add_subnode(fdt, xscom_offset, name);
> +    _FDT(offset);
> +    g_free(name);
> +
> +    _FDT((fdt_setprop(fdt, offset, "reg", reg, sizeof(reg))));
> +
> +    _FDT((fdt_setprop_cell(fdt, offset, "ibm,phb-index", phb->phb_id)));
> +    _FDT((fdt_setprop_cell(fdt, offset, "ibm,chip-id", phb->chip_id)));
> +    _FDT((fdt_setprop(fdt, offset, "compatible", compat,
> +                      sizeof(compat))));
> +    return 0;
> +}
> +
> +static void pnv_pbcq_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    PnvXScomInterfaceClass *xdc = PNV_XSCOM_INTERFACE_CLASS(klass);
> +
> +    xdc->dt_xscom = pnv_pbcq_dt_xscom;
> +
> +    dc->realize = pnv_pbcq_realize;
> +}
> +
> +static const TypeInfo pnv_pbcq_type_info = {
> +    .name          = TYPE_PNV_PBCQ,
> +    .parent        = TYPE_DEVICE,
> +    .instance_size = sizeof(PnvPBCQState),
> +    .class_init    = pnv_pbcq_class_init,
> +    .interfaces    = (InterfaceInfo[]) {
> +        { TYPE_PNV_XSCOM_INTERFACE },
> +        { }
> +    }
> +};
> +
> +static void pnv_pbcq_register_types(void)
> +{
> +    type_register_static(&pnv_pbcq_type_info);
> +}
> +
> +type_init(pnv_pbcq_register_types)
> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> index 7401ffe5b01c..77e92d5e2e66 100644
> --- a/hw/ppc/pnv.c
> +++ b/hw/ppc/pnv.c
> @@ -44,6 +44,7 @@
>  #include "hw/isa/isa.h"
>  #include "hw/char/serial.h"
>  #include "hw/timer/mc146818rtc.h"
> +#include "hw/pci/msi.h"
>  
>  #include <libfdt.h>
>  
> @@ -555,6 +556,9 @@ static void pnv_init(MachineState *machine)
>      int i;
>      char *chip_typename;
>  
> +    /* MSIs are supported on this platform */
> +    msi_nonbroken = true;
> +
>      /* allocate RAM */
>      if (machine->ram_size < (1 * G_BYTE)) {
>          warn_report("skiboot may not work with < 1GB of RAM");
> @@ -725,6 +729,8 @@ static Object *pnv_chip_power9_intc_create(PnvChip *chip, Object *child,
>  static void pnv_chip_power8_instance_init(Object *obj)
>  {
>      Pnv8Chip *chip8 = PNV8_CHIP(obj);
> +    PnvChipClass *pcc = PNV_CHIP_GET_CLASS(obj);
> +    int i;
>  
>      object_initialize(&chip8->psi, sizeof(chip8->psi), TYPE_PNV_PSI);
>      object_property_add_child(obj, "psi", OBJECT(&chip8->psi), NULL);
> @@ -740,6 +746,12 @@ static void pnv_chip_power8_instance_init(Object *obj)
>      object_property_add_child(obj, "occ", OBJECT(&chip8->occ), NULL);
>      object_property_add_const_link(OBJECT(&chip8->occ), "psi",
>                                     OBJECT(&chip8->psi), &error_abort);
> +
> +    for (i = 0; i < pcc->num_phbs; i++) {
> +        object_initialize(&chip8->phbs[i], sizeof(chip8->phbs[i]),
> +                          TYPE_PNV_PHB3);
> +        object_property_add_child(obj, "phb[*]", OBJECT(&chip8->phbs[i]), NULL);
> +    }
>  }
>  
>  static void pnv_chip_icp_realize(Pnv8Chip *chip8, Error **errp)
> @@ -780,6 +792,7 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
>      PnvChip *chip = PNV_CHIP(dev);
>      Pnv8Chip *chip8 = PNV8_CHIP(dev);
>      Error *local_err = NULL;
> +    int i;
>  
>      pcc->parent_realize(dev, &local_err);
>      if (local_err) {
> @@ -817,6 +830,33 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
>          return;
>      }
>      pnv_xscom_add_subregion(chip, PNV_XSCOM_OCC_BASE, &chip8->occ.xscom_regs);
> +
> +    /* Create the PHB3 controllers */
> +    for (i = 0; i < pcc->num_phbs; i++) {
> +        PnvPHB3 *phb = &chip8->phbs[i];
> +        PnvPBCQState *pbcq = &phb->pbcq;
> +
> +        object_property_set_int(OBJECT(phb), i, "phb-id", &error_fatal);
> +        object_property_set_int(OBJECT(phb), chip->chip_id, "chip-id",
> +                                &error_fatal);
> +        object_property_set_bool(OBJECT(phb), true, "realized", &local_err);
> +        if (local_err) {
> +            error_propagate(errp, local_err);
> +            return;
> +        }
> +        qdev_set_parent_bus(DEVICE(phb), sysbus_get_default());
> +
> +        /* Populate the XSCOM address space. */
> +        pnv_xscom_add_subregion(chip,
> +                                PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
> +                                &pbcq->xscom_nest_regs);
> +        pnv_xscom_add_subregion(chip,
> +                                PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
> +                                &pbcq->xscom_pci_regs);
> +        pnv_xscom_add_subregion(chip,
> +                                PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
> +                                &pbcq->xscom_spci_regs);
> +    }
>  }
>  
>  static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
> @@ -827,6 +867,7 @@ static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER8E;
>      k->chip_cfam_id = 0x221ef04980000000ull;  /* P8 Murano DD2.1 */
>      k->cores_mask = POWER8E_CORE_MASK;
> +    k->num_phbs = 2;
>      k->core_pir = pnv_chip_core_pir_p8;
>      k->intc_create = pnv_chip_power8_intc_create;
>      k->isa_create = pnv_chip_power8_isa_create;
> @@ -845,6 +886,7 @@ static void pnv_chip_power8_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER8;
>      k->chip_cfam_id = 0x220ea04980000000ull; /* P8 Venice DD2.0 */
>      k->cores_mask = POWER8_CORE_MASK;
> +    k->num_phbs = 3;
>      k->core_pir = pnv_chip_core_pir_p8;
>      k->intc_create = pnv_chip_power8_intc_create;
>      k->isa_create = pnv_chip_power8_isa_create;
> @@ -863,6 +905,7 @@ static void pnv_chip_power8nvl_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER8NVL;
>      k->chip_cfam_id = 0x120d304980000000ull;  /* P8 Naples DD1.0 */
>      k->cores_mask = POWER8_CORE_MASK;
> +    k->num_phbs = 4;
>      k->core_pir = pnv_chip_core_pir_p8;
>      k->intc_create = pnv_chip_power8_intc_create;
>      k->isa_create = pnv_chip_power8nvl_isa_create;
> @@ -897,6 +940,7 @@ static void pnv_chip_power9_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER9;
>      k->chip_cfam_id = 0x220d104900008000ull; /* P9 Nimbus DD2.0 */
>      k->cores_mask = POWER9_CORE_MASK;
> +    k->num_phbs = 0;
>      k->core_pir = pnv_chip_core_pir_p9;
>      k->intc_create = pnv_chip_power9_intc_create;
>      k->isa_create = pnv_chip_power9_isa_create;
> @@ -1047,14 +1091,24 @@ static void pnv_chip_class_init(ObjectClass *klass, void *data)
>  static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
>  {
>      PnvMachineState *pnv = PNV_MACHINE(xi);
> -    int i;
> +    int i, j;
>  
>      for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
>          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
>  
>          if (ics_valid_irq(&chip8->psi.ics, irq)) {
>              return &chip8->psi.ics;
>          }
> +        for (j = 0; j < pcc->num_phbs; j++) {
> +            if (ics_valid_irq(&chip8->phbs[j].lsis, irq)) {
> +                return &chip8->phbs[j].lsis;
> +            }
> +            if (ics_valid_irq(ICS_BASE(&chip8->phbs[j].msis), irq)) {
> +                return ICS_BASE(&chip8->phbs[j].msis);
> +            }
> +        }
>      }
>      return NULL;
>  }
> @@ -1062,11 +1116,18 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
>  static void pnv_ics_resend(XICSFabric *xi)
>  {
>      PnvMachineState *pnv = PNV_MACHINE(xi);
> -    int i;
> +    int i, j;
>  
>      for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
>          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
> +
>          ics_resend(&chip8->psi.ics);
> +        for (j = 0; j < pcc->num_phbs; j++) {
> +            ics_resend(&chip8->phbs[j].lsis);
> +            ics_resend(ICS_BASE(&chip8->phbs[j].msis));
> +        }
>      }
>  }
>  
> @@ -1097,7 +1158,7 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
>                                 Monitor *mon)
>  {
>      PnvMachineState *pnv = PNV_MACHINE(obj);
> -    int i;
> +    int i, j;
>      CPUState *cs;
>  
>      CPU_FOREACH(cs) {
> @@ -1107,8 +1168,16 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
>      }
>  
>      for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
>          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
> +
>          ics_pic_print_info(&chip8->psi.ics, mon);
> +
> +        for (j = 0; j < pcc->num_phbs; j++) {
> +            ics_pic_print_info(&chip8->phbs[j].lsis, mon);
> +            ics_pic_print_info(ICS_BASE(&chip8->phbs[j].msis), mon);
> +        }
>      }
>  }
>  
> diff --git a/hw/ppc/pnv_xscom.c b/hw/ppc/pnv_xscom.c
> index 46fae41f32b0..83f05d06053f 100644
> --- a/hw/ppc/pnv_xscom.c
> +++ b/hw/ppc/pnv_xscom.c
> @@ -252,7 +252,11 @@ int pnv_dt_xscom(PnvChip *chip, void *fdt, int root_offset)
>      args.fdt = fdt;
>      args.xscom_offset = xscom_offset;
>  
> -    object_child_foreach(OBJECT(chip), xscom_dt_child, &args);
> +    /* Some PnvXScomInterface objects lie a bit deeper (PnvPBCQState)
> +     * than the first layer, so we need to loop on the whole object
> +     * hierarchy to catch them
> +     */
> +    object_child_foreach_recursive(OBJECT(chip), xscom_dt_child, &args);
>      return 0;
>  }
>  
> diff --git a/hw/pci-host/Makefile.objs b/hw/pci-host/Makefile.objs
> index 6d6597c06563..1b5b782bd986 100644
> --- a/hw/pci-host/Makefile.objs
> +++ b/hw/pci-host/Makefile.objs
> @@ -19,3 +19,4 @@ common-obj-$(CONFIG_PCI_GENERIC) += gpex.o
>  common-obj-$(CONFIG_PCI_XILINX) += xilinx-pcie.o
>  
>  common-obj-$(CONFIG_PCI_DESIGNWARE) += designware.o
> +obj-$(CONFIG_POWERNV) += pnv_phb3.o pnv_phb3_pbcq.o pnv_phb3_msi.o
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 1/2] " Cédric Le Goater
  2018-07-09  7:22   ` Cédric Le Goater
@ 2018-07-18  6:12   ` David Gibson
  2018-07-18  8:03     ` Benjamin Herrenschmidt
                       ` (4 more replies)
  2018-07-26  9:03   ` Cédric Le Goater
  2 siblings, 5 replies; 29+ messages in thread
From: David Gibson @ 2018-07-18  6:12 UTC (permalink / raw)
  To: Cédric Le Goater
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

[-- Attachment #1: Type: text/plain, Size: 108257 bytes --]

On Thu, Jun 28, 2018 at 10:36:32AM +0200, Cédric Le Goater wrote:
> From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> 
> This is a model of the PCIe Host Bridge (PHB3) found on a Power8
> processor. It includes the PowerBus logic interface (PBCQ), IOMMU
> support, PCIe root complex, the XICS MSI and LSI interrupt sources.
> 
> The Power8 processor comes in different flavors: Venice, Murano,
> Naple, each having a different number of PHBs. All are initialized but
> the machine only needs one PCI bus to plug a couple of devices (net,
> storage, usbs). The other busses are provided to test complex
> configurations. Some platforms, like the Firestone, can also couple
> PHBs on the first chip to provide more bandwidth but this is too
> specific to model in QEMU.
> 
> No default device layout is provided and PCI devices can be added on
> any of the available PCI busses (pci.0..2 on a Power8 chip) using
> command line options such as :
> 
>   -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.0,addr=0x2
>   -netdev bridge,id=net0,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=hostnet0
> 
>   -device megasas,id=scsi0,bus=pci.0,addr=0x1
>   -drive file=$disk,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none
>   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2
> 
> Multi chip is supported, each chip adding its set of PHB3 controllers
> and its PCI busses. The model doesn't emulate the EEH error handling
> (and may never do).
> 
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> [clg: rewrote the QOM models
>       misc fixes
>       lots of love and care]
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
>  default-configs/ppc64-softmmu.mak   |    1 +
>  include/hw/pci-host/pnv_phb3.h      |  161 +++++
>  include/hw/pci-host/pnv_phb3_regs.h |  467 ++++++++++++++
>  include/hw/ppc/pnv.h                |   22 +
>  include/hw/ppc/pnv_xscom.h          |    9 +
>  include/hw/ppc/xics.h               |    1 +
>  hw/intc/xics.c                      |    2 +-
>  hw/pci-host/pnv_phb3.c              | 1219 +++++++++++++++++++++++++++++++++++
>  hw/pci-host/pnv_phb3_msi.c          |  316 +++++++++
>  hw/pci-host/pnv_phb3_pbcq.c         |  347 ++++++++++
>  hw/ppc/pnv.c                        |   75 ++-
>  hw/ppc/pnv_xscom.c                  |    6 +-
>  hw/pci-host/Makefile.objs           |    1 +
>  13 files changed, 2622 insertions(+), 5 deletions(-)
>  create mode 100644 include/hw/pci-host/pnv_phb3.h
>  create mode 100644 include/hw/pci-host/pnv_phb3_regs.h
>  create mode 100644 hw/pci-host/pnv_phb3.c
>  create mode 100644 hw/pci-host/pnv_phb3_msi.c
>  create mode 100644 hw/pci-host/pnv_phb3_pbcq.c
> 
> diff --git a/default-configs/ppc64-softmmu.mak b/default-configs/ppc64-softmmu.mak
> index b94af6c7c62a..deebba2b044a 100644
> --- a/default-configs/ppc64-softmmu.mak
> +++ b/default-configs/ppc64-softmmu.mak
> @@ -9,6 +9,7 @@ CONFIG_IPMI=y
>  CONFIG_IPMI_LOCAL=y
>  CONFIG_IPMI_EXTERN=y
>  CONFIG_ISA_IPMI_BT=y
> +CONFIG_PCIE_PORT=y
>  
>  # For pSeries
>  CONFIG_PSERIES=y
> diff --git a/include/hw/pci-host/pnv_phb3.h b/include/hw/pci-host/pnv_phb3.h
> new file mode 100644
> index 000000000000..459994b751ca
> --- /dev/null
> +++ b/include/hw/pci-host/pnv_phb3.h
> @@ -0,0 +1,161 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +
> +#ifndef PCI_HOST_PNV_PHB3_H
> +#define PCI_HOST_PNV_PHB3_H
> +
> +#include "hw/pci/pci_host.h"
> +#include "hw/ppc/xics.h"
> +
> +typedef struct PnvPHB3 PnvPHB3;
> +typedef struct PnvChip PnvChip;
> +
> +/*
> + * PHB3 XICS Source for MSIs
> + */
> +#define TYPE_PHB3_MSI "phb3-msi"
> +#define PHB3_MSI(obj) OBJECT_CHECK(Phb3MsiState, (obj), TYPE_PHB3_MSI)
> +
> +#define PHB3_MAX_MSI     2048
> +
> +typedef struct Phb3MsiState {
> +    ICSState ics;
> +
> +    PnvPHB3 *phb;
> +    uint64_t rba[PHB3_MAX_MSI / 64];
> +    uint32_t rba_sum;
> +} Phb3MsiState;
> +
> +void pnv_phb3_msi_update_config(Phb3MsiState *msis, uint32_t base,
> +                                uint32_t count);
> +void pnv_phb3_msi_send(Phb3MsiState *msis, uint64_t addr, uint16_t data,
> +                       int32_t dev_pe);
> +void pnv_phb3_msi_ffi(Phb3MsiState *msis, uint64_t val);
> +
> +
> +/* We have one such address space wrapper per possible device
> + * under the PHB since they need to be assigned statically at
> + * qemu device creation time. The relationship to a PE is done
> + * later dynamically. This means we can potentially create a lot
> + * of these guys. Q35 stores them as some kind of radix tree but
> + * we never really need to do fast lookups so instead we simply
> + * keep a QLIST of them for now, we can add the radix if needed
> + * later on.
> + *
> + * We do cache the PE number to speed things up a bit though.
> + */
> +typedef struct PnvPhb3DMASpace {
> +    PCIBus *bus;
> +    uint8_t devfn;
> +    int pe_num;         /* Cached PE number */
> +#define PHB_INVALID_PE (-1)
> +    PnvPHB3 *phb;
> +    AddressSpace dma_as;
> +    IOMMUMemoryRegion dma_mr;
> +    MemoryRegion msi32_mr;
> +    MemoryRegion msi64_mr;
> +    bool msi32_mapped;
> +    bool msi64_mapped;
> +    QLIST_ENTRY(PnvPhb3DMASpace) list;
> +} PnvPhb3DMASpace;
> +
> +/*
> + * PHB3 Power Bus Common Queue
> + */
> +#define TYPE_PNV_PBCQ "pnv-pbcq"
> +#define PNV_PBCQ(obj) OBJECT_CHECK(PnvPBCQState, (obj), TYPE_PNV_PBCQ)
> +
> +typedef struct PnvPBCQState {
> +    DeviceState parent;
> +
> +    uint32_t nest_xbase;
> +    uint32_t spci_xbase;
> +    uint32_t pci_xbase;
> +#define PBCQ_NEST_REGS_COUNT    0x46
> +#define PBCQ_PCI_REGS_COUNT     0x15
> +#define PBCQ_SPCI_REGS_COUNT    0x5
> +
> +    uint64_t nest_regs[PBCQ_NEST_REGS_COUNT];
> +    uint64_t spci_regs[PBCQ_SPCI_REGS_COUNT];
> +    uint64_t pci_regs[PBCQ_PCI_REGS_COUNT];
> +    MemoryRegion mmbar0;
> +    MemoryRegion mmbar1;
> +    MemoryRegion phbbar;
> +    bool mmio0_mapped;
> +    bool mmio1_mapped;
> +    bool phb_mapped;
> +    uint64_t mmio0_base;
> +    uint64_t mmio0_size;
> +    uint64_t mmio1_base;
> +    uint64_t mmio1_size;
> +    PnvPHB3 *phb;
> +
> +    MemoryRegion xscom_nest_regs;
> +    MemoryRegion xscom_pci_regs;
> +    MemoryRegion xscom_spci_regs;
> +} PnvPBCQState;
> +
> +/*
> + * PHB3 PCI Host Bridge for PowerNV machines (POWER8)
> + */
> +#define TYPE_PNV_PHB3 "pnv-phb3"
> +#define PNV_PHB3(obj) OBJECT_CHECK(PnvPHB3, (obj), TYPE_PNV_PHB3)
> +
> +#define PNV_PHB3_NUM_M64      16
> +#define PNV_PHB3_NUM_REGS     (0x1000 >> 3)
> +#define PNV_PHB3_NUM_LSI      8
> +#define PNV_PHB3_NUM_PE       256
> +
> +#define PCI_MMIO_TOTAL_SIZE   (0x1ull << 60)
> +
> +struct PnvPHB3 {
> +    PCIHostState parent_obj;
> +
> +    uint32_t chip_id;
> +    uint32_t phb_id;
> +    char bus_path[8];
> +
> +    uint64_t regs[PNV_PHB3_NUM_REGS];
> +    MemoryRegion mr_regs;
> +
> +    MemoryRegion mr_m32;
> +    MemoryRegion mr_m64[PNV_PHB3_NUM_M64];
> +    bool regs_mapped;
> +    bool m32_mapped;
> +    bool m64_mapped[PNV_PHB3_NUM_M64];
> +    MemoryRegion pci_mmio;
> +    MemoryRegion pci_io;
> +
> +    uint64_t ioda_LIST[8];
> +    uint64_t ioda_LXIVT[8];
> +    uint64_t ioda_TVT[512];
> +    uint64_t ioda_M64BT[16];
> +    uint64_t ioda_MDT[256];
> +    uint64_t ioda_PEEV[4];
> +
> +    uint32_t total_irq;
> +    ICSState lsis;
> +    Phb3MsiState msis;
> +
> +    PnvPBCQState pbcq;
> +
> +    QLIST_HEAD(, PnvPhb3DMASpace) dma_spaces;
> +};
> +
> +#define TYPE_PNV_PHB3_RC "pnv-phb3-rc"
> +
> +#define TYPE_PNV_PHB3_ROOT_BUS "pnv-phb3-root-bus"
> +
> +uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size);
> +void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size);
> +void pnv_phb3_update_regions(PnvPHB3 *phb);
> +void pnv_phb3_remap_irqs(PnvPHB3 *phb);
> +
> +
> +#endif /* PCI_HOST_PNV_PHB3_H */
> diff --git a/include/hw/pci-host/pnv_phb3_regs.h b/include/hw/pci-host/pnv_phb3_regs.h
> new file mode 100644
> index 000000000000..4ddbb5559eb5
> --- /dev/null
> +++ b/include/hw/pci-host/pnv_phb3_regs.h
> @@ -0,0 +1,467 @@
> +/* Copyright (c) 2013-2018, IBM Corporation.
> + *
> + * Licensed under the Apache License, Version 2.0 (the "License");
> + * you may not use this file except in compliance with the License.
> + * You may obtain a copy of the License at
> + *
> + *      http://www.apache.org/licenses/LICENSE-2.0
> + *
> + * Unless required by applicable law or agreed to in writing, software
> + * distributed under the License is distributed on an "AS IS" BASIS,
> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> + * implied.
> + *
> + * See the License for the specific language governing permissions and
> + * limitations under the License.
> + */
> +
> +#ifndef PCI_HOST_PNV_PHB3_REGS_H
> +#define PCI_HOST_PNV_PHB3_REGS_H
> +
> +/*
> + * PBCQ XSCOM registers
> + */
> +
> +#define PBCQ_NEST_IRSN_COMPARE  0x1a
> +#define PBCQ_NEST_IRSN_COMP           PPC_BITMASK(0, 18)
> +#define PBCQ_NEST_IRSN_MASK     0x1b
> +#define PBCQ_NEST_LSI_SRC_ID    0x1f
> +#define   PBCQ_NEST_LSI_SRC           PPC_BITMASK(0, 7)
> +#define PBCQ_NEST_REGS_COUNT    0x46
> +#define PBCQ_NEST_MMIO_BAR0     0x40
> +#define PBCQ_NEST_MMIO_BAR1     0x41
> +#define PBCQ_NEST_PHB_BAR       0x42
> +#define PBCQ_NEST_MMIO_MASK0    0x43
> +#define PBCQ_NEST_MMIO_MASK1    0x44
> +#define PBCQ_NEST_BAR_EN        0x45
> +#define   PBCQ_NEST_BAR_EN_MMIO0    PPC_BIT(0)
> +#define   PBCQ_NEST_BAR_EN_MMIO1    PPC_BIT(1)
> +#define   PBCQ_NEST_BAR_EN_PHB      PPC_BIT(2)
> +#define   PBCQ_NEST_BAR_EN_IRSN_RX  PPC_BIT(3)
> +#define   PBCQ_NEST_BAR_EN_IRSN_TX  PPC_BIT(4)
> +
> +#define PBCQ_PCI_REGS_COUNT     0x15
> +#define PBCQ_PCI_BAR2           0x0b
> +
> +#define PBCQ_SPCI_REGS_COUNT    0x5
> +#define PBCQ_SPCI_ASB_ADDR      0x0
> +#define PBCQ_SPCI_ASB_STATUS    0x1
> +#define PBCQ_SPCI_ASB_DATA      0x2
> +#define PBCQ_SPCI_AIB_CAPP_EN   0x3
> +#define PBCQ_SPCI_CAPP_SEC_TMR  0x4
> +
> +/*
> + * PHB MMIO registers
> + */
> +
> +/* PHB Fundamental register set A */
> +#define PHB_LSI_SOURCE_ID               0x100
> +#define   PHB_LSI_SRC_ID                PPC_BITMASK(5, 12)
> +#define PHB_DMA_CHAN_STATUS             0x110
> +#define   PHB_DMA_CHAN_ANY_ERR          PPC_BIT(27)
> +#define   PHB_DMA_CHAN_ANY_ERR1         PPC_BIT(28)
> +#define   PHB_DMA_CHAN_ANY_FREEZE       PPC_BIT(29)
> +#define PHB_CPU_LOADSTORE_STATUS        0x120
> +#define   PHB_CPU_LS_ANY_ERR            PPC_BIT(27)
> +#define   PHB_CPU_LS_ANY_ERR1           PPC_BIT(28)
> +#define   PHB_CPU_LS_ANY_FREEZE         PPC_BIT(29)
> +#define PHB_DMA_MSI_NODE_ID             0x128
> +#define   PHB_DMAMSI_NID_FIXED          PPC_BIT(0)
> +#define   PHB_DMAMSI_NID_MASK           PPC_BITMASK(24, 31)
> +#define   PHB_DMAMSI_NID_LSH            PPC_BITLSHIFT(31)
> +#define PHB_CONFIG_DATA                 0x130
> +#define PHB_LOCK0                       0x138
> +#define PHB_CONFIG_ADDRESS              0x140
> +#define   PHB_CA_ENABLE                 PPC_BIT(0)
> +#define   PHB_CA_BUS_MASK               PPC_BITMASK(4, 11)
> +#define   PHB_CA_BUS_LSH                PPC_BITLSHIFT(11)
> +#define   PHB_CA_DEV_MASK               PPC_BITMASK(12, 16)
> +#define   PHB_CA_DEV_LSH                PPC_BITLSHIFT(16)
> +#define   PHB_CA_FUNC_MASK              PPC_BITMASK(17, 19)
> +#define   PHB_CA_FUNC_LSH               PPC_BITLSHIFT(19)
> +#define   PHB_CA_REG_MASK               PPC_BITMASK(20, 31)
> +#define   PHB_CA_REG_LSH                PPC_BITLSHIFT(31)
> +#define   PHB_CA_PE_MASK                PPC_BITMASK(40, 47)
> +#define   PHB_CA_PE_LSH                 PPC_BITLSHIFT(47)
> +#define PHB_LOCK1                       0x148
> +#define PHB_IVT_BAR                     0x150
> +#define   PHB_IVT_BAR_ENABLE            PPC_BIT(0)
> +#define   PHB_IVT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 48)
> +#define   PHB_IVT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(48)
> +#define   PHB_IVT_LENGTH_MASK           PPC_BITMASK(52, 63)
> +#define   PHB_IVT_LENGTH_ADDRESS_LSH    PPC_BITLSHIFT(63)
> +#define PHB_RBA_BAR                     0x158
> +#define   PHB_RBA_BAR_ENABLE            PPC_BIT(0)
> +#define   PHB_RBA_BASE_ADDRESS_MASK     PPC_BITMASK(14, 55)
> +#define   PHB_RBA_BASE_ADDRESS_LSH      PPC_BITLSHIFT(55)
> +#define PHB_PHB3_CONFIG                 0x160
> +#define   PHB_PHB3C_64B_TCE_EN          PPC_BIT(2)
> +#define   PHB_PHB3C_32BIT_MSI_EN        PPC_BIT(8)
> +#define   PHB_PHB3C_64BIT_MSI_EN        PPC_BIT(14)
> +#define   PHB_PHB3C_M32_EN              PPC_BIT(16)
> +#define PHB_RTT_BAR                     0x168
> +#define   PHB_RTT_BAR_ENABLE            PPC_BIT(0)
> +#define   PHB_RTT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 46)
> +#define   PHB_RTT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(46)
> +#define PHB_PELTV_BAR                   0x188
> +#define   PHB_PELTV_BAR_ENABLE          PPC_BIT(0)
> +#define   PHB_PELTV_BASE_ADDRESS_MASK   PPC_BITMASK(14, 50)
> +#define   PHB_PELTV_BASE_ADDRESS_LSH    PPC_BITLSHIFT(50)
> +#define PHB_M32_BASE_ADDR               0x190
> +#define PHB_M32_BASE_MASK               0x198
> +#define PHB_M32_START_ADDR              0x1a0
> +#define PHB_PEST_BAR                    0x1a8
> +#define   PHB_PEST_BAR_ENABLE           PPC_BIT(0)
> +#define   PHB_PEST_BASE_ADDRESS_MASK    PPC_BITMASK(14, 51)
> +#define   PHB_PEST_BASE_ADDRESS_LSH     PPC_BITLSHIFT(51)
> +#define PHB_M64_UPPER_BITS              0x1f0
> +#define PHB_INTREP_TIMER                0x1f8
> +#define PHB_DMARD_SYNC                  0x200
> +#define PHB_RTC_INVALIDATE              0x208
> +#define   PHB_RTC_INVALIDATE_ALL        PPC_BIT(0)
> +#define   PHB_RTC_INVALIDATE_RID_MASK   PPC_BITMASK(16, 31)
> +#define   PHB_RTC_INVALIDATE_RID_LSH    PPC_BITLSHIFT(31)
> +#define PHB_TCE_KILL                    0x210
> +#define   PHB_TCE_KILL_ALL              PPC_BIT(0)
> +#define PHB_TCE_SPEC_CTL                0x218
> +#define PHB_IODA_ADDR                   0x220
> +#define   PHB_IODA_AD_AUTOINC           PPC_BIT(0)
> +#define   PHB_IODA_AD_TSEL              PPC_BITMASK(11, 15)
> +#define   PHB_IODA_AD_TADR              PPC_BITMASK(55, 63)
> +#define PHB_IODA_DATA0                  0x228
> +#define PHB_FFI_REQUEST                 0x238
> +#define   PHB_FFI_LOCK_CLEAR            PPC_BIT(3)
> +#define   PHB_FFI_REQUEST_ISN_MASK      PPC_BITMASK(49, 59)
> +#define   PHB_FFI_REQUEST_ISN_LSH       PPC_BITLSHIFT(59)
> +#define PHB_FFI_LOCK                    0x240
> +#define   PHB_FFI_LOCK_STATE            PPC_BIT(0)
> +#define PHB_XIVE_UPDATE                 0x248 /* Broken in DD1 */
> +#define PHB_PHB3_GEN_CAP                0x250
> +#define PHB_PHB3_TCE_CAP                0x258
> +#define PHB_PHB3_IRQ_CAP                0x260
> +#define PHB_PHB3_EEH_CAP                0x268
> +#define PHB_IVC_INVALIDATE              0x2a0
> +#define   PHB_IVC_INVALIDATE_ALL        PPC_BIT(0)
> +#define   PHB_IVC_INVALIDATE_SID_MASK   PPC_BITMASK(16, 31)
> +#define   PHB_IVC_INVALIDATE_SID_LSH    PPC_BITLSHIFT(31)
> +#define PHB_IVC_UPDATE                  0x2a8
> +#define   PHB_IVC_UPDATE_ENABLE_P       PPC_BIT(0)
> +#define   PHB_IVC_UPDATE_ENABLE_Q       PPC_BIT(1)
> +#define   PHB_IVC_UPDATE_ENABLE_SERVER  PPC_BIT(2)
> +#define   PHB_IVC_UPDATE_ENABLE_PRI     PPC_BIT(3)
> +#define   PHB_IVC_UPDATE_ENABLE_GEN     PPC_BIT(4)
> +#define   PHB_IVC_UPDATE_ENABLE_CON     PPC_BIT(5)
> +#define   PHB_IVC_UPDATE_GEN_MATCH_MASK PPC_BITMASK(6, 7)
> +#define   PHB_IVC_UPDATE_GEN_MATCH_LSH  PPC_BITLSHIFT(7)
> +#define   PHB_IVC_UPDATE_SERVER_MASK    PPC_BITMASK(8, 23)
> +#define   PHB_IVC_UPDATE_SERVER_LSH     PPC_BITLSHIFT(23)
> +#define   PHB_IVC_UPDATE_PRI_MASK       PPC_BITMASK(24, 31)
> +#define   PHB_IVC_UPDATE_PRI_LSH        PPC_BITLSHIFT(31)
> +#define   PHB_IVC_UPDATE_GEN_MASK       PPC_BITMASK(32, 33)
> +#define   PHB_IVC_UPDATE_GEN_LSH        PPC_BITLSHIFT(33)
> +#define   PHB_IVC_UPDATE_P_MASK         PPC_BITMASK(34, 34)
> +#define   PHB_IVC_UPDATE_P_LSH          PPC_BITLSHIFT(34)
> +#define   PHB_IVC_UPDATE_Q_MASK         PPC_BITMASK(35, 35)
> +#define   PHB_IVC_UPDATE_Q_LSH          PPC_BITLSHIFT(35)
> +#define   PHB_IVC_UPDATE_SID_MASK       PPC_BITMASK(48, 63)
> +#define   PHB_IVC_UPDATE_SID_LSH        PPC_BITLSHIFT(63)
> +#define PHB_PAPR_ERR_INJ_CTL            0x2b0
> +#define   PHB_PAPR_ERR_INJ_CTL_INB      PPC_BIT(0)
> +#define   PHB_PAPR_ERR_INJ_CTL_OUTB     PPC_BIT(1)
> +#define   PHB_PAPR_ERR_INJ_CTL_STICKY   PPC_BIT(2)
> +#define   PHB_PAPR_ERR_INJ_CTL_CFG      PPC_BIT(3)
> +#define   PHB_PAPR_ERR_INJ_CTL_RD       PPC_BIT(4)
> +#define   PHB_PAPR_ERR_INJ_CTL_WR       PPC_BIT(5)
> +#define   PHB_PAPR_ERR_INJ_CTL_FREEZE   PPC_BIT(6)
> +#define PHB_PAPR_ERR_INJ_ADDR           0x2b8
> +#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_MASK       PPC_BITMASK(16, 63)
> +#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_LSH        PPC_BITLSHIFT(63)
> +#define PHB_PAPR_ERR_INJ_MASK           0x2c0
> +#define   PHB_PAPR_ERR_INJ_MASK_CFG_MASK        PPC_BITMASK(4, 11)
> +#define   PHB_PAPR_ERR_INJ_MASK_CFG_LSH         PPC_BITLSHIFT(11)
> +#define   PHB_PAPR_ERR_INJ_MASK_MMIO_MASK       PPC_BITMASK(16, 63)
> +#define   PHB_PAPR_ERR_INJ_MASK_MMIO_LSH        PPC_BITLSHIFT(63)
> +#define PHB_ETU_ERR_SUMMARY             0x2c8
> +
> +/*  UTL registers */
> +#define UTL_SYS_BUS_CONTROL             0x400
> +#define UTL_STATUS                      0x408
> +#define UTL_SYS_BUS_AGENT_STATUS        0x410
> +#define UTL_SYS_BUS_AGENT_ERR_SEVERITY  0x418
> +#define UTL_SYS_BUS_AGENT_IRQ_EN        0x420
> +#define UTL_SYS_BUS_BURST_SZ_CONF       0x440
> +#define UTL_REVISION_ID                 0x448
> +#define UTL_BCLK_DOMAIN_DBG1            0x460
> +#define UTL_BCLK_DOMAIN_DBG2            0x468
> +#define UTL_BCLK_DOMAIN_DBG3            0x470
> +#define UTL_BCLK_DOMAIN_DBG4            0x478
> +#define UTL_BCLK_DOMAIN_DBG5            0x480
> +#define UTL_BCLK_DOMAIN_DBG6            0x488
> +#define UTL_OUT_POST_HDR_BUF_ALLOC      0x4c0
> +#define UTL_OUT_POST_DAT_BUF_ALLOC      0x4d0
> +#define UTL_IN_POST_HDR_BUF_ALLOC       0x4e0
> +#define UTL_IN_POST_DAT_BUF_ALLOC       0x4f0
> +#define UTL_OUT_NP_BUF_ALLOC            0x500
> +#define UTL_IN_NP_BUF_ALLOC             0x510
> +#define UTL_PCIE_TAGS_ALLOC             0x520
> +#define UTL_GBIF_READ_TAGS_ALLOC        0x530
> +#define UTL_PCIE_PORT_CONTROL           0x540
> +#define UTL_PCIE_PORT_STATUS            0x548
> +#define UTL_PCIE_PORT_ERROR_SEV         0x550
> +#define UTL_PCIE_PORT_IRQ_EN            0x558
> +#define UTL_RC_STATUS                   0x560
> +#define UTL_RC_ERR_SEVERITY             0x568
> +#define UTL_RC_IRQ_EN                   0x570
> +#define UTL_EP_STATUS                   0x578
> +#define UTL_EP_ERR_SEVERITY             0x580
> +#define UTL_EP_ERR_IRQ_EN               0x588
> +#define UTL_PCI_PM_CTRL1                0x590
> +#define UTL_PCI_PM_CTRL2                0x598
> +#define UTL_GP_CTL1                     0x5a0
> +#define UTL_GP_CTL2                     0x5a8
> +#define UTL_PCLK_DOMAIN_DBG1            0x5b0
> +#define UTL_PCLK_DOMAIN_DBG2            0x5b8
> +#define UTL_PCLK_DOMAIN_DBG3            0x5c0
> +#define UTL_PCLK_DOMAIN_DBG4            0x5c8
> +
> +/* PCI-E Stack registers */
> +#define PHB_PCIE_SYSTEM_CONFIG          0x600
> +#define PHB_PCIE_BUS_NUMBER             0x608
> +#define PHB_PCIE_SYSTEM_TEST            0x618
> +#define PHB_PCIE_LINK_MANAGEMENT        0x630
> +#define   PHB_PCIE_LM_LINK_ACTIVE       PPC_BIT(8)
> +#define PHB_PCIE_DLP_TRAIN_CTL          0x640
> +#define   PHB_PCIE_DLP_TCTX_DISABLE     PPC_BIT(1)
> +#define   PHB_PCIE_DLP_TCRX_DISABLED    PPC_BIT(16)
> +#define   PHB_PCIE_DLP_INBAND_PRESENCE  PPC_BIT(19)
> +#define   PHB_PCIE_DLP_TC_DL_LINKUP     PPC_BIT(21)
> +#define   PHB_PCIE_DLP_TC_DL_PGRESET    PPC_BIT(22)
> +#define   PHB_PCIE_DLP_TC_DL_LINKACT    PPC_BIT(23)
> +#define PHB_PCIE_SLOP_LOOPBACK_STATUS   0x648
> +#define PHB_PCIE_SYS_LINK_INIT          0x668
> +#define PHB_PCIE_UTL_CONFIG             0x670
> +#define PHB_PCIE_DLP_CONTROL            0x678
> +#define PHB_PCIE_UTL_ERRLOG1            0x680
> +#define PHB_PCIE_UTL_ERRLOG2            0x688
> +#define PHB_PCIE_UTL_ERRLOG3            0x690
> +#define PHB_PCIE_UTL_ERRLOG4            0x698
> +#define PHB_PCIE_DLP_ERRLOG1            0x6a0
> +#define PHB_PCIE_DLP_ERRLOG2            0x6a8
> +#define PHB_PCIE_DLP_ERR_STATUS         0x6b0
> +#define PHB_PCIE_DLP_ERR_COUNTERS       0x6b8
> +#define PHB_PCIE_UTL_ERR_INJECT         0x6c0
> +#define PHB_PCIE_TLDLP_ERR_INJECT       0x6c8
> +#define PHB_PCIE_LANE_EQ_CNTL0          0x6d0
> +#define PHB_PCIE_LANE_EQ_CNTL1          0x6d8
> +#define PHB_PCIE_LANE_EQ_CNTL2          0x6e0
> +#define PHB_PCIE_LANE_EQ_CNTL3          0x6e8
> +#define PHB_PCIE_STRAPPING              0x700
> +
> +/* Fundamental register set B */
> +#define PHB_VERSION                     0x800
> +#define PHB_RESET                       0x808
> +#define PHB_CONTROL                     0x810
> +#define   PHB_CTRL_IVE_128_BYTES        PPC_BIT(24)
> +#define PHB_AIB_RX_CRED_INIT_TIMER      0x818
> +#define PHB_AIB_RX_CMD_CRED             0x820
> +#define PHB_AIB_RX_DATA_CRED            0x828
> +#define PHB_AIB_TX_CMD_CRED             0x830
> +#define PHB_AIB_TX_DATA_CRED            0x838
> +#define PHB_AIB_TX_CHAN_MAPPING         0x840
> +#define PHB_AIB_TAG_ENABLE              0x858
> +#define PHB_AIB_FENCE_CTRL              0x860
> +#define PHB_TCE_TAG_ENABLE              0x868
> +#define PHB_TCE_WATERMARK               0x870
> +#define PHB_TIMEOUT_CTRL1               0x878
> +#define PHB_TIMEOUT_CTRL2               0x880
> +#define PHB_QUIESCE_DMA_G               0x888
> +#define PHB_AIB_TAG_STATUS              0x900
> +#define PHB_TCE_TAG_STATUS              0x908
> +
> +/* FIR & Error registers */
> +#define PHB_LEM_FIR_ACCUM               0xc00
> +#define PHB_LEM_FIR_AND_MASK            0xc08
> +#define PHB_LEM_FIR_OR_MASK             0xc10
> +#define PHB_LEM_ERROR_MASK              0xc18
> +#define PHB_LEM_ERROR_AND_MASK          0xc20
> +#define PHB_LEM_ERROR_OR_MASK           0xc28
> +#define PHB_LEM_ACTION0                 0xc30
> +#define PHB_LEM_ACTION1                 0xc38
> +#define PHB_LEM_WOF                     0xc40
> +#define PHB_ERR_STATUS                  0xc80
> +#define PHB_ERR1_STATUS                 0xc88
> +#define PHB_ERR_INJECT                  0xc90
> +#define PHB_ERR_LEM_ENABLE              0xc98
> +#define PHB_ERR_IRQ_ENABLE              0xca0
> +#define PHB_ERR_FREEZE_ENABLE           0xca8
> +#define PHB_ERR_AIB_FENCE_ENABLE        0xcb0
> +#define PHB_ERR_LOG_0                   0xcc0
> +#define PHB_ERR_LOG_1                   0xcc8
> +#define PHB_ERR_STATUS_MASK             0xcd0
> +#define PHB_ERR1_STATUS_MASK            0xcd8
> +
> +#define PHB_OUT_ERR_STATUS              0xd00
> +#define PHB_OUT_ERR1_STATUS             0xd08
> +#define PHB_OUT_ERR_INJECT              0xd10
> +#define PHB_OUT_ERR_LEM_ENABLE          0xd18
> +#define PHB_OUT_ERR_IRQ_ENABLE          0xd20
> +#define PHB_OUT_ERR_FREEZE_ENABLE       0xd28
> +#define PHB_OUT_ERR_AIB_FENCE_ENABLE    0xd30
> +#define PHB_OUT_ERR_LOG_0               0xd40
> +#define PHB_OUT_ERR_LOG_1               0xd48
> +#define PHB_OUT_ERR_STATUS_MASK         0xd50
> +#define PHB_OUT_ERR1_STATUS_MASK        0xd58
> +
> +#define PHB_INA_ERR_STATUS              0xd80
> +#define PHB_INA_ERR1_STATUS             0xd88
> +#define PHB_INA_ERR_INJECT              0xd90
> +#define PHB_INA_ERR_LEM_ENABLE          0xd98
> +#define PHB_INA_ERR_IRQ_ENABLE          0xda0
> +#define PHB_INA_ERR_FREEZE_ENABLE       0xda8
> +#define PHB_INA_ERR_AIB_FENCE_ENABLE    0xdb0
> +#define PHB_INA_ERR_LOG_0               0xdc0
> +#define PHB_INA_ERR_LOG_1               0xdc8
> +#define PHB_INA_ERR_STATUS_MASK         0xdd0
> +#define PHB_INA_ERR1_STATUS_MASK        0xdd8
> +
> +#define PHB_INB_ERR_STATUS              0xe00
> +#define PHB_INB_ERR1_STATUS             0xe08
> +#define PHB_INB_ERR_INJECT              0xe10
> +#define PHB_INB_ERR_LEM_ENABLE          0xe18
> +#define PHB_INB_ERR_IRQ_ENABLE          0xe20
> +#define PHB_INB_ERR_FREEZE_ENABLE       0xe28
> +#define PHB_INB_ERR_AIB_FENCE_ENABLE    0xe30
> +#define PHB_INB_ERR_LOG_0               0xe40
> +#define PHB_INB_ERR_LOG_1               0xe48
> +#define PHB_INB_ERR_STATUS_MASK         0xe50
> +#define PHB_INB_ERR1_STATUS_MASK        0xe58
> +
> +/* Performance monitor & Debug registers */
> +#define PHB_TRACE_CONTROL               0xf80
> +#define PHB_PERFMON_CONFIG              0xf88
> +#define PHB_PERFMON_CTR0                0xf90
> +#define PHB_PERFMON_CTR1                0xf98
> +#define PHB_PERFMON_CTR2                0xfa0
> +#define PHB_PERFMON_CTR3                0xfa8
> +#define PHB_HOTPLUG_OVERRIDE            0xfb0
> +#define   PHB_HPOVR_FORCE_RESAMPLE      PPC_BIT(9)
> +#define   PHB_HPOVR_PRESENCE_A          PPC_BIT(10)
> +#define   PHB_HPOVR_PRESENCE_B          PPC_BIT(11)
> +#define   PHB_HPOVR_LINK_ACTIVE         PPC_BIT(12)
> +#define   PHB_HPOVR_LINK_BIFURCATED     PPC_BIT(13)
> +#define   PHB_HPOVR_LINK_LANE_SWAPPED   PPC_BIT(14)
> +
> +/*
> + * IODA2 on-chip tables
> + */
> +
> +#define IODA2_TBL_LIST          1
> +#define IODA2_TBL_LXIVT         2
> +#define IODA2_TBL_IVC_CAM       3
> +#define IODA2_TBL_RBA           4
> +#define IODA2_TBL_RCAM          5
> +#define IODA2_TBL_MRT           6
> +#define IODA2_TBL_PESTA         7
> +#define IODA2_TBL_PESTB         8
> +#define IODA2_TBL_TVT           9
> +#define IODA2_TBL_TCAM          10
> +#define IODA2_TBL_TDR           11
> +#define IODA2_TBL_M64BT         16
> +#define IODA2_TBL_M32DT         17
> +#define IODA2_TBL_PEEV          20
> +
> +/* LXIVT */
> +#define IODA2_LXIVT_SERVER              PPC_BITMASK(8, 23)
> +#define IODA2_LXIVT_PRIORITY            PPC_BITMASK(24, 31)
> +#define IODA2_LXIVT_NODE_ID             PPC_BITMASK(56, 63)
> +
> +/* IVT */
> +#define IODA2_IVT_SERVER                PPC_BITMASK(0, 23)
> +#define IODA2_IVT_PRIORITY              PPC_BITMASK(24, 31)
> +#define IODA2_IVT_GEN                   PPC_BITMASK(37, 38)
> +#define IODA2_IVT_P                     PPC_BITMASK(39, 39)
> +#define IODA2_IVT_Q                     PPC_BITMASK(47, 47)
> +#define IODA2_IVT_PE                    PPC_BITMASK(48, 63)
> +
> +/* TVT */
> +#define IODA2_TVT_TABLE_ADDR            PPC_BITMASK(0, 47)
> +#define IODA2_TVT_NUM_LEVELS            PPC_BITMASK(48, 50)
> +#define   IODA2_TVE_1_LEVEL     0
> +#define   IODA2_TVE_2_LEVELS    1
> +#define   IODA2_TVE_3_LEVELS    2
> +#define   IODA2_TVE_4_LEVELS    3
> +#define   IODA2_TVE_5_LEVELS    4
> +#define IODA2_TVT_TCE_TABLE_SIZE        PPC_BITMASK(51, 55)
> +#define IODA2_TVT_IO_PSIZE              PPC_BITMASK(59, 63)
> +
> +/* PESTA */
> +#define IODA2_PESTA_MMIO_FROZEN         PPC_BIT(0)
> +
> +/* PESTB */
> +#define IODA2_PESTB_DMA_STOPPED         PPC_BIT(0)
> +
> +/* M32DT */
> +#define IODA2_M32DT_PE_MASK             PPC_BITMASK(8, 15)
> +#define IODA2_M32DT_PE_LSH              PPC_BITLSHIFT(15)
> +
> +/* M64BT */
> +#define IODA2_M64BT_ENABLE              PPC_BIT(0)
> +#define IODA2_M64BT_SINGLE_PE           PPC_BIT(1)
> +#define IODA2_M64BT_BASE                PPC_BITMASK(2, 31)
> +#define IODA2_M64BT_MASK                PPC_BITMASK(34, 63)
> +#define IODA2_M64BT_SINGLE_BASE_MASK    PPC_BITMASK(2, 26)
> +#define IODA2_M64BT_SINGLE_BASE_LSH     PPC_BITLSHIFT(26)
> +#define IODA2_M64BT_PE_HI_MASK          PPC_BITMASK(27, 31)
> +#define IODA2_M64BT_PE_HI_LSH           PPC_BITLSHIFT(31)
> +#define IODA2_M64BT_SINGLE_MASK_MASK    PPC_BITMASK(34, 58)
> +#define IODA2_M64BT_SINGLE_MASK_LSH     PPC_BITLSHIFT(58)
> +#define IODA2_M64BT_PE_LOW_MASK         PPC_BITMASK(59, 63)
> +#define IODA2_M64BT_PE_LOW_LSH          PPC_BITLSHIFT(63)
> +
> +/*
> + * IODA2 in-memory tables
> + */
> +
> +/* PEST
> + *
> + * 2x8 bytes entries, PEST0 and PEST1
> + */
> +
> +#define IODA2_PEST0_MMIO_CAUSE          PPC_BIT(2)
> +#define IODA2_PEST0_CFG_READ            PPC_BIT(3)
> +#define IODA2_PEST0_CFG_WRITE           PPC_BIT(4)
> +#define IODA2_PEST0_TTYPE_MASK          PPC_BITMASK(5, 7)
> +#define IODA2_PEST0_TTYPE_LSH           PPC_BITLSHIFT(7)
> +#define   PEST_TTYPE_DMA_WRITE          0
> +#define   PEST_TTYPE_MSI                1
> +#define   PEST_TTYPE_DMA_READ           2
> +#define   PEST_TTYPE_DMA_READ_RESP      3
> +#define   PEST_TTYPE_MMIO_LOAD          4
> +#define   PEST_TTYPE_MMIO_STORE         5
> +#define   PEST_TTYPE_OTHER              7
> +#define IODA2_PEST0_CA_RETURN           PPC_BIT(8)
> +#define IODA2_PEST0_UTL_RTOS_TIMEOUT    PPC_BIT(8) /* Same bit as CA return */
> +#define IODA2_PEST0_UR_RETURN           PPC_BIT(9)
> +#define IODA2_PEST0_UTL_NONFATAL        PPC_BIT(10)
> +#define IODA2_PEST0_UTL_FATAL           PPC_BIT(11)
> +#define IODA2_PEST0_PARITY_UE           PPC_BIT(13)
> +#define IODA2_PEST0_UTL_CORRECTABLE     PPC_BIT(14)
> +#define IODA2_PEST0_UTL_INTERRUPT       PPC_BIT(15)
> +#define IODA2_PEST0_MMIO_XLATE          PPC_BIT(16)
> +#define IODA2_PEST0_IODA2_ERROR         PPC_BIT(16) /* Same bit as MMIO xlate */
> +#define IODA2_PEST0_TCE_PAGE_FAULT      PPC_BIT(18)
> +#define IODA2_PEST0_TCE_ACCESS_FAULT    PPC_BIT(19)
> +#define IODA2_PEST0_DMA_RESP_TIMEOUT    PPC_BIT(20)
> +#define IODA2_PEST0_AIB_SIZE_INVALID    PPC_BIT(21)
> +#define IODA2_PEST0_LEM_BIT_MASK        PPC_BITMASK(26, 31)
> +#define IODA2_PEST0_LEM_BIT_LSH         PPC_BITLSHIFT(31)
> +#define IODA2_PEST0_RID_MASK            PPC_BITMASK(32, 47)
> +#define IODA2_PEST0_RID_LSH             PPC_BITLSHIFT(47)
> +#define IODA2_PEST0_MSI_DATA_MASK       PPC_BITMASK(48, 63)
> +#define IODA2_PEST0_MSI_DATA_LSH        PPC_BITLSHIFT(63)
> +
> +#define IODA2_PEST1_FAIL_ADDR_MASK      PPC_BITMASK(3, 63)
> +#define IODA2_PEST1_FAIL_ADDR_LSH       PPC_BITLSHIFT(63)
> +
> +
> +#endif /* PCI_HOST_PNV_PHB3_REGS_H */
> diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
> index 86d5f54e5459..61af65fc0b76 100644
> --- a/include/hw/ppc/pnv.h
> +++ b/include/hw/ppc/pnv.h
> @@ -25,6 +25,7 @@
>  #include "hw/ppc/pnv_lpc.h"
>  #include "hw/ppc/pnv_psi.h"
>  #include "hw/ppc/pnv_occ.h"
> +#include "hw/pci-host/pnv_phb3.h"
>  
>  #define TYPE_PNV_CHIP "pnv-chip"
>  #define PNV_CHIP(obj) OBJECT_CHECK(PnvChip, (obj), TYPE_PNV_CHIP)
> @@ -72,6 +73,9 @@ typedef struct Pnv8Chip {
>      PnvLpcController lpc;
>      PnvPsi       psi;
>      PnvOCC       occ;
> +
> +#define PNV8_CHIP_PHB3_MAX 4
> +    PnvPHB3      phbs[PNV8_CHIP_PHB3_MAX];
>  } Pnv8Chip;
>  
>  #define TYPE_PNV9_CHIP "pnv9-chip"
> @@ -92,6 +96,7 @@ typedef struct PnvChipClass {
>      PnvChipType  chip_type;
>      uint64_t     chip_cfam_id;
>      uint64_t     cores_mask;
> +    uint32_t     num_phbs;
>  
>      hwaddr       xscom_base;
>  
> @@ -171,6 +176,23 @@ static inline bool pnv_is_power9(PnvMachineState *pnv)
>      return pnv_chip_is_power9(pnv->chips[0]);
>  }
>  
> +/*
> + * This is used by devices created on the command line to find a chip
> + * on which to attach to.
> + */
> +static inline PnvChip *pnv_get_chip(PnvMachineState *pnv, uint chip_id)
> +{
> +    int i;
> +
> +    for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
> +        if (chip->chip_id == chip_id) {
> +            return chip;
> +        }
> +    }
> +    return NULL;
> +}
> +
>  #define PNV_FDT_ADDR          0x01000000
>  #define PNV_TIMEBASE_FREQ     512000000ULL
>  
> diff --git a/include/hw/ppc/pnv_xscom.h b/include/hw/ppc/pnv_xscom.h
> index 255b26a5aaf6..e7ebffa5cf8a 100644
> --- a/include/hw/ppc/pnv_xscom.h
> +++ b/include/hw/ppc/pnv_xscom.h
> @@ -73,6 +73,15 @@ typedef struct PnvXScomInterfaceClass {
>  #define PNV_XSCOM_OCC_BASE        0x0066000
>  #define PNV_XSCOM_OCC_SIZE        0x6000
>  
> +#define PNV_XSCOM_PBCQ_NEST_BASE  0x2012000
> +#define PNV_XSCOM_PBCQ_NEST_SIZE  0x46
> +
> +#define PNV_XSCOM_PBCQ_PCI_BASE   0x9012000
> +#define PNV_XSCOM_PBCQ_PCI_SIZE   0x15
> +
> +#define PNV_XSCOM_PBCQ_SPCI_BASE  0x9013c00
> +#define PNV_XSCOM_PBCQ_SPCI_SIZE  0x5
> +
>  extern void pnv_xscom_realize(PnvChip *chip, Error **errp);
>  extern int pnv_dt_xscom(PnvChip *chip, void *fdt, int offset);
>  
> diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> index 6ac8a9392da6..966a996c2eac 100644
> --- a/include/hw/ppc/xics.h
> +++ b/include/hw/ppc/xics.h
> @@ -194,6 +194,7 @@ void icp_set_mfrr(ICPState *icp, uint8_t mfrr);
>  uint32_t icp_accept(ICPState *ss);
>  uint32_t icp_ipoll(ICPState *ss, uint32_t *mfrr);
>  void icp_eoi(ICPState *icp, uint32_t xirr);
> +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority);

Hrm... are you sure you need to expose this?

>  void ics_simple_write_xive(ICSState *ics, int nr, int server,
>                             uint8_t priority, uint8_t saved_priority);
> diff --git a/hw/intc/xics.c b/hw/intc/xics.c
> index b9f1a3c97214..59e2a5217dcc 100644
> --- a/hw/intc/xics.c
> +++ b/hw/intc/xics.c
> @@ -230,7 +230,7 @@ void icp_eoi(ICPState *icp, uint32_t xirr)
>      }
>  }
>  
> -static void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
> +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
>  {
>      ICPState *icp = xics_icp_get(ics->xics, server);
>  
> diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
> new file mode 100644
> index 000000000000..e11993e0e1fb
> --- /dev/null
> +++ b/hw/pci-host/pnv_phb3.c
> @@ -0,0 +1,1219 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qemu/log.h"
> +#include "qapi/visitor.h"
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "target/ppc/cpu.h"
> +#include "hw/pci-host/pnv_phb3_regs.h"
> +#include "hw/pci-host/pnv_phb3.h"
> +#include "hw/pci/pci_bridge.h"
> +#include "hw/pci/pci_bus.h"
> +#include "hw/pci/pcie_port.h"
> +#include "hw/ppc/pnv.h"
> +
> +static PCIDevice *pnv_phb3_find_cfg_dev(PnvPHB3 *phb)
> +{
> +    PCIHostState *pci = PCI_HOST_BRIDGE(phb);
> +    uint64_t addr = phb->regs[PHB_CONFIG_ADDRESS >> 3];
> +    uint8_t bus, devfn;
> +
> +    if (!(addr >> 63)) {
> +        return NULL;
> +    }
> +    bus = (addr >> 52) & 0xff;
> +    devfn = (addr >> 44) & 0xff;
> +
> +    return pci_find_device(pci->bus, bus, devfn);
> +}
> +
> +/*
> + * The CONFIG_DATA register expects little endian accesses, but as the
> + * region is big endian, we have to swap the value.
> + */
> +static void pnv_phb3_config_write(PnvPHB3 *phb, unsigned off,
> +                                  unsigned size, uint64_t val)
> +{
> +    uint32_t cfg_addr, limit;
> +    PCIDevice *pdev;
> +
> +    pdev = pnv_phb3_find_cfg_dev(phb);
> +    if (!pdev) {
> +        return;
> +    }
> +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xfff;
> +    cfg_addr |= off;

This looks weird - there are callers of this that appear to have low
bits in 'off', then you're ORing it with overlapping low bits.

> +    limit = pci_config_size(pdev);
> +    if (limit <= cfg_addr) {
> +        /* conventional pci device can be behind pcie-to-pci bridge.
> +           256 <= addr < 4K has no effects. */
> +        return;
> +    }
> +    switch (size) {
> +    case 1:
> +        break;
> +    case 2:
> +        val = bswap16(val);
> +        break;
> +    case 4:
> +        val = bswap32(val);
> +        break;
> +    default:
> +        g_assert_not_reached();
> +    }
> +    pci_host_config_write_common(pdev, cfg_addr, limit, val, size);
> +}
> +
> +static uint64_t pnv_phb3_config_read(PnvPHB3 *phb, unsigned off,
> +                                     unsigned size)
> +{
> +    uint32_t cfg_addr, limit;
> +    PCIDevice *pdev;
> +    uint64_t val;
> +
> +    pdev = pnv_phb3_find_cfg_dev(phb);
> +    if (!pdev) {
> +        return ~0ull;
> +    }
> +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xffc;
> +    cfg_addr |= off;

This looks better, should it be 0xffc in the write path as well?

> +    limit = pci_config_size(pdev);
> +    if (limit <= cfg_addr) {
> +        /* conventional pci device can be behind pcie-to-pci bridge.
> +           256 <= addr < 4K has no effects. */
> +        return ~0ull;
> +    }
> +    val = pci_host_config_read_common(pdev, cfg_addr, limit, size);
> +    switch (size) {
> +    case 1:
> +        return val;
> +    case 2:
> +        return bswap16(val);
> +    case 4:
> +        return bswap32(val);
> +    default:
> +        g_assert_not_reached();
> +    }
> +}
> +
> +static void pnv_phb3_check_m32(PnvPHB3 *phb)
> +{
> +    uint64_t base, start, size;
> +    MemoryRegion *parent;
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    if (phb->m32_mapped) {
> +        memory_region_del_subregion(phb->mr_m32.container, &phb->mr_m32);
> +        phb->m32_mapped = false;

Could you use memory_region_set_enabled() rather than having to add
and delete the subregion and keep track of it in this separate
variable?

> +    }
> +
> +    /* Disabled ? move on with life ... */

Trivial: "nothing to do" seems to be the standard way to express this.
Even trivialler: usual English rules don't put a space before '?' or
'!' punctuation.

> +    if (!(phb->regs[PHB_PHB3_CONFIG >> 3] & PHB_PHB3C_M32_EN)) {
> +        return;
> +    }
> +
> +    /* Grab geometry from registers */
> +    base = phb->regs[PHB_M32_BASE_ADDR >> 3];
> +    start = phb->regs[PHB_M32_START_ADDR >> 3];
> +    size = ~(phb->regs[PHB_M32_BASE_MASK >> 3] | 0xfffc000000000000ull) + 1;
> +
> +    /* Check if it matches an enabled MMIO region in the PBCQ */
> +    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
> +        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
> +        parent = &pbcq->mmbar0;
> +        base -= pbcq->mmio0_base;
> +    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
> +        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
> +        parent = &pbcq->mmbar1;
> +        base -= pbcq->mmio1_base;
> +    } else {
> +        return;
> +    }
> +
> +    /* Create alias */
> +    memory_region_init_alias(&phb->mr_m32, OBJECT(phb), "phb3-m32",
> +                             &phb->pci_mmio, start, size);
> +    memory_region_add_subregion(parent, base, &phb->mr_m32);
> +    phb->m32_mapped = true;
> +}
> +
> +static void pnv_phb3_check_m64(PnvPHB3 *phb, uint32_t index)
> +{
> +    uint64_t base, start, size, m64;
> +    MemoryRegion *parent;
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    if (phb->m64_mapped[index]) {
> +        /* Should we destroy it in RCU friendly way... ? */
> +        memory_region_del_subregion(phb->mr_m64[index].container,
> +                                    &phb->mr_m64[index]);
> +        phb->m64_mapped[index] = false;
> +    }
> +
> +    /* Get table entry */
> +    m64 = phb->ioda_M64BT[index];
> +
> +    /* Disabled ? move on with life ... */
> +    if (!(m64 & IODA2_M64BT_ENABLE)) {
> +        return;
> +    }
> +
> +    /* Grab geometry from registers */
> +    base = GETFIELD(IODA2_M64BT_BASE, m64) << 20;
> +    if (m64 & IODA2_M64BT_SINGLE_PE) {
> +        base &= ~0x1ffffffull;
> +    }
> +    size = GETFIELD(IODA2_M64BT_MASK, m64) << 20;
> +    size |= 0xfffc000000000000ull;
> +    size = ~size + 1;
> +    start = base | (phb->regs[PHB_M64_UPPER_BITS >> 3]);
> +
> +    /* Check if it matches an enabled MMIO region in the PBCQ */
> +    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
> +        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
> +        parent = &pbcq->mmbar0;
> +        base -= pbcq->mmio0_base;
> +    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
> +        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
> +        parent = &pbcq->mmbar1;
> +        base -= pbcq->mmio1_base;
> +    } else {
> +        return;
> +    }
> +
> +    /* Create alias */
> +    memory_region_init_alias(&phb->mr_m64[index], OBJECT(phb), "phb3-m64",
> +                             &phb->pci_mmio, start, size);
> +    memory_region_add_subregion(parent, base, &phb->mr_m64[index]);
> +    phb->m64_mapped[index] = true;
> +}
> +
> +static void pnv_phb3_check_all_m64s(PnvPHB3 *phb)
> +{
> +    uint64_t i;
> +
> +    for (i = 0; i < PNV_PHB3_NUM_M64; i++) {
> +        pnv_phb3_check_m64(phb, i);
> +    }
> +}
> +
> +static void pnv_phb3_lxivt_write(PnvPHB3 *phb, unsigned idx, uint64_t val)
> +{
> +    ICSState *ics = &phb->lsis;
> +    uint8_t server, prio;
> +
> +    phb->ioda_LXIVT[idx] = val & (IODA2_LXIVT_SERVER |
> +                                  IODA2_LXIVT_PRIORITY |
> +                                  IODA2_LXIVT_NODE_ID);
> +    server = GETFIELD(IODA2_LXIVT_SERVER, val);
> +    prio = GETFIELD(IODA2_LXIVT_PRIORITY, val);
> +
> +    /*
> +     * The low order 2 bits are the link pointer (Type II interrupts).

I don't think we've currently implemented link pointers in our xics
code.  Do we need to do that?

> +     * Shift back to get a valid IRQ server.
> +     */
> +    server >>= 2;
> +
> +    ics_simple_write_xive(ics, idx, server, prio, prio);
> +}
> +
> +static uint64_t *pnv_phb3_ioda_access(PnvPHB3 *phb,
> +                                      unsigned *out_table, unsigned *out_idx)
> +{
> +    uint64_t adreg = phb->regs[PHB_IODA_ADDR >> 3];
> +    unsigned int index = GETFIELD(PHB_IODA_AD_TADR, adreg);
> +    unsigned int table = GETFIELD(PHB_IODA_AD_TSEL, adreg);
> +    unsigned int mask;
> +    uint64_t *tptr = NULL;
> +
> +    switch (table) {
> +    case IODA2_TBL_LIST:
> +        tptr = phb->ioda_LIST;
> +        mask = 7;

I'd suggest hex for the masks.

> +        break;
> +    case IODA2_TBL_LXIVT:
> +        tptr = phb->ioda_LXIVT;
> +        mask = 7;
> +        break;
> +    case IODA2_TBL_IVC_CAM:
> +    case IODA2_TBL_RBA:
> +        mask = 31;
> +        break;
> +    case IODA2_TBL_RCAM:
> +        mask = 63;
> +        break;
> +    case IODA2_TBL_MRT:
> +        mask = 7;
> +        break;
> +    case IODA2_TBL_PESTA:
> +    case IODA2_TBL_PESTB:
> +        mask = 255;
> +        break;
> +    case IODA2_TBL_TVT:
> +        tptr = phb->ioda_TVT;
> +        mask = 511;
> +        break;
> +    case IODA2_TBL_TCAM:
> +    case IODA2_TBL_TDR:
> +        mask = 63;
> +        break;
> +    case IODA2_TBL_M64BT:
> +        tptr = phb->ioda_M64BT;
> +        mask = 15;
> +        break;
> +    case IODA2_TBL_M32DT:
> +        tptr = phb->ioda_MDT;
> +        mask = 255;
> +        break;
> +    case IODA2_TBL_PEEV:
> +        tptr = phb->ioda_PEEV;
> +        mask = 3;
> +        break;
> +    default:
> +        return NULL;
> +    }
> +    index &= mask;

Do we want to silently mask, rather than logging an error if there's
an access out of bounds for a particular table?

> +    if (out_idx) {
> +        *out_idx = index;
> +    }
> +    if (out_table) {
> +        *out_table = table;
> +    }
> +    if (adreg & PHB_IODA_AD_AUTOINC) {
> +        index = (index + 1) & mask;
> +        adreg = SETFIELD(PHB_IODA_AD_TADR, adreg, index);
> +    }
> +    if (tptr) {
> +        tptr += index;
> +    }
> +    phb->regs[PHB_IODA_ADDR >> 3] = adreg;
> +    return tptr;
> +}
> +
> +static uint64_t pnv_phb3_ioda_read(PnvPHB3 *phb)
> +{
> +        unsigned table;
> +        uint64_t *tptr;
> +
> +        tptr = pnv_phb3_ioda_access(phb, &table, NULL);
> +        if (!tptr) {
> +            /* Return 0 on unsupported tables, not ff's */
> +            return 0;
> +        }
> +        return *tptr;
> +}
> +
> +static void pnv_phb3_ioda_write(PnvPHB3 *phb, uint64_t val)
> +{
> +        unsigned table, idx;
> +        uint64_t *tptr;
> +
> +        tptr = pnv_phb3_ioda_access(phb, &table, &idx);
> +        if (!tptr) {
> +            return;
> +        }
> +
> +        /* Handle side effects */
> +        switch (table) {
> +        case IODA2_TBL_LXIVT:
> +            pnv_phb3_lxivt_write(phb, idx, val);
> +            break;
> +        case IODA2_TBL_M64BT:
> +            *tptr = val;
> +            pnv_phb3_check_m64(phb, idx);
> +            break;
> +        default:
> +            *tptr = val;
> +        }
> +}
> +
> +/* This is called whenever the PHB LSI, MSI source ID register or
> + * the PBCQ irq filters are written.
> + */
> +void pnv_phb3_remap_irqs(PnvPHB3 *phb)
> +{
> +    ICSState *ics = &phb->lsis;
> +    uint32_t local, global, count, mask, comp;
> +    uint64_t baren;
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    /* First check if we are enabled. Unlike real HW we don't separate TX and RX
> +     * so we enable if both are set
> +     */
> +    baren = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
> +    if (!(baren & PBCQ_NEST_BAR_EN_IRSN_RX) ||
> +        !(baren & PBCQ_NEST_BAR_EN_IRSN_TX)) {
> +        ics->offset = 0;
> +        return;
> +    }
> +
> +    /* Grab local LSI source ID */
> +    local = GETFIELD(PHB_LSI_SRC_ID, phb->regs[PHB_LSI_SOURCE_ID >> 3]) << 3;
> +
> +    /* Grab global one and compare */
> +    global = GETFIELD(PBCQ_NEST_LSI_SRC,
> +                      pbcq->nest_regs[PBCQ_NEST_LSI_SRC_ID]) << 3;
> +    if (global != local) {
> +        /* This happens during initialization, let's come back when we
> +         * are properly configured
> +         */
> +        ics->offset = 0;
> +        return;
> +    }
> +
> +    /* Get the base on the powerbus */
> +    comp = GETFIELD(PBCQ_NEST_IRSN_COMP,
> +                    pbcq->nest_regs[PBCQ_NEST_IRSN_COMPARE]);
> +    mask = GETFIELD(PBCQ_NEST_IRSN_COMP,
> +                    pbcq->nest_regs[PBCQ_NEST_IRSN_MASK]);
> +    count = ((~mask) + 1) & 0x7ffff;
> +    phb->total_irq = count;
> +
> +    /* Sanity checks */
> +    if ((global + PNV_PHB3_NUM_LSI) > count) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "LSIs out of reach: LSI base=%d total irq=%d",
> +                      global, count);
> +    }
> +
> +    if (count > 2048) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "More interrupts than supported: %d", count);
> +    }
> +
> +    if ((comp & mask) != comp) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "IRQ compare bits not in mask: comp=0x%x mask=0x%x",
> +                      comp, mask);
> +        comp &= mask;
> +    }
> +    /* Setup LSI offset */
> +    ics->offset = comp + global;

Oh.. changing ICS offset at runtime.  I hadn't considered that case..

> +    /* Setup MSI offset */
> +    pnv_phb3_msi_update_config(&phb->msis, comp, count - PNV_PHB3_NUM_LSI);
> +}
> +
> +static void pnv_phb3_lsi_src_id_write(PnvPHB3 *phb, uint64_t val)
> +{
> +    /* Sanitize content */
> +    val &= PHB_LSI_SRC_ID;
> +    phb->regs[PHB_LSI_SOURCE_ID >> 3] = val;
> +    pnv_phb3_remap_irqs(phb);
> +}
> +
> +static void pnv_phb3_rtc_invalidate(PnvPHB3 *phb, uint64_t val)
> +{
> +    PnvPhb3DMASpace *ds;
> +
> +    /* Always invalidate all for now ... */
> +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> +        ds->pe_num = PHB_INVALID_PE;
> +    }
> +}
> +
> +
> +static void pnv_phb3_update_msi_regions(PnvPhb3DMASpace *ds)
> +{
> +    uint64_t cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
> +
> +    if (cfg & PHB_PHB3C_32BIT_MSI_EN) {
> +        if (!ds->msi32_mapped) {
> +            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        0xffff0000, &ds->msi32_mr);
> +            ds->msi32_mapped = true;
> +        }
> +    } else {
> +        if (ds->msi32_mapped) {
> +            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        &ds->msi32_mr);
> +            ds->msi32_mapped = false;
> +        }
> +    }
> +
> +    if (cfg & PHB_PHB3C_64BIT_MSI_EN) {
> +        if (!ds->msi64_mapped) {
> +            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        (1ull << 60), &ds->msi64_mr);
> +            ds->msi64_mapped = true;
> +        }
> +    } else {
> +        if (ds->msi64_mapped) {
> +            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
> +                                        &ds->msi64_mr);
> +            ds->msi64_mapped = false;
> +        }
> +    }
> +}
> +
> +static void pnv_phb3_update_all_msi_regions(PnvPHB3 *phb)
> +{
> +    PnvPhb3DMASpace *ds;
> +
> +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> +        pnv_phb3_update_msi_regions(ds);
> +    }
> +}
> +
> +void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size)
> +{
> +    PnvPHB3 *phb = opaque;
> +    bool changed;
> +
> +    /* Special case configuration data */
> +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> +        pnv_phb3_config_write(phb, off & 0x3, size, val);
> +        return;
> +    }
> +
> +    /* Other registers are 64-bit only */
> +    if (size != 8 || off & 0x7) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> +                      off, size);
> +        return;
> +    }
> +
> +    /* Handle masking */
> +    switch (off) {
> +    case PHB_M64_UPPER_BITS:

Couldn't you handle this in the switch below - it should be ok to
momentarily have the invalid bits set in your reg case, as long as you
mask them before applying the side-effects, yes?

That said... shouldn't you filter our invalid or read-only regs before
updating the cache?

> +        val &= 0xfffc000000000000ull;
> +        break;
> +    }
> +
> +    /* Record whether it changed */
> +    changed = phb->regs[off >> 3] != val;
> +
> +    /* Store in register cache first */
> +    phb->regs[off >> 3] = val;
> +
> +    /* Handle side effects */
> +    switch (off) {
> +    case PHB_PHB3_CONFIG:
> +        if (changed) {
> +            pnv_phb3_update_all_msi_regions(phb);
> +        }
> +        /* fall through */
> +    case PHB_M32_BASE_ADDR:
> +    case PHB_M32_BASE_MASK:
> +    case PHB_M32_START_ADDR:
> +        if (changed) {
> +            pnv_phb3_check_m32(phb);
> +        }
> +        break;
> +    case PHB_M64_UPPER_BITS:
> +        if (changed) {
> +            pnv_phb3_check_all_m64s(phb);
> +        }
> +        break;
> +    case PHB_LSI_SOURCE_ID:
> +        if (changed) {
> +            pnv_phb3_lsi_src_id_write(phb, val);
> +        }
> +        break;
> +
> +    /* IODA table accesses */
> +    case PHB_IODA_DATA0:
> +        pnv_phb3_ioda_write(phb, val);
> +        break;
> +
> +    /* RTC invalidation */
> +    case PHB_RTC_INVALIDATE:
> +        pnv_phb3_rtc_invalidate(phb, val);
> +        break;
> +
> +    /* FFI request */
> +    case PHB_FFI_REQUEST:
> +        pnv_phb3_msi_ffi(&phb->msis, val);
> +        break;
> +
> +    /* Silent simple writes */
> +    case PHB_CONFIG_ADDRESS:
> +    case PHB_IODA_ADDR:
> +    case PHB_TCE_KILL:
> +    case PHB_TCE_SPEC_CTL:
> +    case PHB_PEST_BAR:
> +    case PHB_PELTV_BAR:
> +    case PHB_RTT_BAR:
> +    case PHB_RBA_BAR:
> +    case PHB_IVT_BAR:
> +    case PHB_FFI_LOCK:
> +        break;
> +
> +    /* Noise on anything else */
> +    default:
> +        qemu_log_mask(LOG_UNIMP, "reg_write 0x%"PRIx64"=%"PRIx64, off, val);
> +    }
> +}
> +
> +uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size)
> +{
> +    PnvPHB3 *phb = opaque;
> +    uint64_t val;
> +
> +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> +        return pnv_phb3_config_read(phb, off & 0x3, size);
> +    }
> +
> +    /* Other registers are 64-bit only */
> +    if (size != 8 || off & 0x7) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> +                      off, size);
> +        return ~0ull;
> +    }
> +
> +    /* Default read from cache */
> +    val = phb->regs[off >> 3];
> +
> +    switch (off) {
> +    /* Simulate venice DD2.0 */
> +    case PHB_VERSION:
> +        return 0x000000a300000005ull;
> +
> +    /* IODA table accesses */
> +    case PHB_IODA_DATA0:
> +        return pnv_phb3_ioda_read(phb);
> +
> +    /* Link training always appears trained */
> +    case PHB_PCIE_DLP_TRAIN_CTL:
> +        return PHB_PCIE_DLP_INBAND_PRESENCE | PHB_PCIE_DLP_TC_DL_LINKACT;
> +
> +    /* FFI Lock */
> +    case PHB_FFI_LOCK:
> +        /* Set lock and return previous value */
> +        phb->regs[off >> 3] |= PHB_FFI_LOCK_STATE;
> +        return val;
> +
> +    /* Silent simple reads */
> +    case PHB_PHB3_CONFIG:
> +    case PHB_M32_BASE_ADDR:
> +    case PHB_M32_BASE_MASK:
> +    case PHB_M32_START_ADDR:
> +    case PHB_CONFIG_ADDRESS:
> +    case PHB_IODA_ADDR:
> +    case PHB_RTC_INVALIDATE:
> +    case PHB_TCE_KILL:
> +    case PHB_TCE_SPEC_CTL:
> +    case PHB_PEST_BAR:
> +    case PHB_PELTV_BAR:
> +    case PHB_RTT_BAR:
> +    case PHB_RBA_BAR:
> +    case PHB_IVT_BAR:
> +    case PHB_M64_UPPER_BITS:
> +        break;
> +
> +    /* Noise on anything else */
> +    default:
> +        qemu_log_mask(LOG_UNIMP, "reg_read 0x%"PRIx64"=%"PRIx64, off, val);
> +    }
> +    return val;
> +}
> +
> +static const MemoryRegionOps pnv_phb3_reg_ops = {
> +    .read = pnv_phb3_reg_read,
> +    .write = pnv_phb3_reg_write,
> +    .valid.min_access_size = 1,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 1,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static int pnv_phb3_map_irq(PCIDevice *pci_dev, int irq_num)
> +{
> +    /* Check that out properly ... */
> +    return irq_num & 3;
> +}
> +
> +static void pnv_phb3_set_irq(void *opaque, int irq_num, int level)
> +{
> +    PnvPHB3 *phb = opaque;
> +
> +    /* LSI only ... */
> +    if (irq_num > 3) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Unknown IRQ to set %d", irq_num);
> +    }
> +    qemu_set_irq(phb->lsis.qirqs[irq_num], level);
> +}
> +
> +static bool pnv_phb3_resolve_pe(PnvPhb3DMASpace *ds)
> +{
> +    uint64_t rtt, addr;
> +    uint16_t rte;
> +    int bus_num;
> +
> +    /* Already resolved ? */
> +    if (ds->pe_num != PHB_INVALID_PE) {
> +        return true;
> +    }
> +
> +    /* We need to lookup the RTT */
> +    rtt = ds->phb->regs[PHB_RTT_BAR >> 3];
> +    if (!(rtt & PHB_RBA_BAR_ENABLE)) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "DMA with RTT BAR disabled !");
> +        /* Set error bits ? fence ? ... */
> +        return false;
> +    }
> +
> +    /* Read RTE */
> +    bus_num = pci_bus_num(ds->bus);
> +    addr = rtt & PHB_RTT_BASE_ADDRESS_MASK;
> +    addr += 2 * ((bus_num << 8) | ds->devfn);
> +    if (dma_memory_read(&address_space_memory, addr, &rte, sizeof(rte))) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read RTT entry at 0x%"PRIx64,
> +                      addr);
> +        /* Set error bits ? fence ? ... */
> +        return false;
> +    }
> +    rte = be16_to_cpu(rte);
> +
> +    /* Fail upon reading of invalid PE# */
> +    if (rte >= PNV_PHB3_NUM_PE) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "RTE for RID 0x%x invalid (%04x)",
> +                      ds->devfn, rte);
> +        /* Set error bits ? fence ? ... */
> +        return false;
> +    }
> +    ds->pe_num = rte;
> +    return true;
> +}
> +
> +static void pnv_phb3_translate_tve(PnvPhb3DMASpace *ds, hwaddr addr,
> +                                   bool is_write, uint64_t tve,
> +                                   IOMMUTLBEntry *tlb)
> +{
> +    uint64_t tta = GETFIELD(IODA2_TVT_TABLE_ADDR, tve);
> +    int32_t  lev = GETFIELD(IODA2_TVT_NUM_LEVELS, tve);
> +    uint32_t tts = GETFIELD(IODA2_TVT_TCE_TABLE_SIZE, tve);
> +    uint32_t tps = GETFIELD(IODA2_TVT_IO_PSIZE, tve);
> +
> +    /* Invalid levels */
> +    if (lev > 4) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Invalid #levels in TVE %d", lev);
> +        return;
> +    }
> +
> +    /* IO Page Size of 0 means untranslated, else use TCEs */
> +    if (tps == 0) {
> +        /* We only support non-translate in top window
> +         * XXX FIX THAT, Venice/Murano support it on bottom window
> +         * above 4G and Naples suports it on everything
> +         */
> +        if (!(tve & PPC_BIT(51))) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                          "xlate for invalid non-translate TVE");
> +            return;
> +        }
> +        /* XXX Handle boundaries */
> +
> +        /* XXX Use 4k pages like q35 ... for now */
> +        tlb->iova = addr & 0xfffffffffffff000ull;
> +        tlb->translated_addr = addr & 0x0003fffffffff000ull;
> +        tlb->addr_mask = 0xfffull;
> +        tlb->perm = IOMMU_RW;
> +    } else {
> +        uint32_t tce_shift, tbl_shift, sh;
> +        uint64_t base, taddr, tce, tce_mask;
> +
> +        /* TVE disabled ? */
> +        if (tts == 0) {
> +            qemu_log_mask(LOG_GUEST_ERROR, "xlate for invalid translated TVE");
> +            return;
> +        }
> +
> +        /* Address bits per bottom level TCE entry */
> +        tce_shift = tps + 11;
> +
> +        /* Address bits per table level */
> +        tbl_shift = tts + 8;
> +
> +        /* Top level table base address */
> +        base = tta << 12;
> +
> +        /* Total shift to first level */
> +        sh = tbl_shift * lev + tce_shift;
> +
> +        /* XXX Multi-level untested */
> +        while ((lev--) >= 0) {
> +            /* Grab the TCE address */
> +            taddr = base | (((addr >> sh) & ((1ul << tbl_shift) - 1)) << 3);
> +            if (dma_memory_read(&address_space_memory, taddr, &tce,
> +                                sizeof(tce))) {
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              "Failed to read TCE at 0x%"PRIx64, taddr);
> +                return;
> +            }
> +            tce = be64_to_cpu(tce);
> +
> +            /* Check permission for indirect TCE */
> +            if ((lev >= 0) && !(tce & 3)) {
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              "Invalid indirect TCE at 0x%"PRIx64, taddr);
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              " xlate %"PRIx64":%c TVE=%"PRIx64,
> +                              addr, is_write ? 'W' : 'R', tve);
> +                qemu_log_mask(LOG_GUEST_ERROR,
> +                              " tta=%"PRIx64" lev=%d tts=%d tps=%d",
> +                              tta, lev, tts, tps);
> +                return;
> +            }
> +            sh -= tbl_shift;
> +            base = tce & ~0xfffull;
> +        }
> +
> +        /* We exit the loop with TCE being the final TCE */
> +        tce_mask = ~((1ull << tce_shift) - 1);
> +        tlb->iova = addr & tce_mask;
> +        tlb->translated_addr = tce & tce_mask;
> +        tlb->addr_mask = ~tce_mask;
> +        tlb->perm = tce & 3;
> +        if ((is_write & !(tce & 2)) || ((!is_write) && !(tce & 1))) {
> +            qemu_log_mask(LOG_GUEST_ERROR, "TCE access fault at 0x%"PRIx64,
> +                          taddr);
> +            qemu_log_mask(LOG_GUEST_ERROR, " xlate %"PRIx64":%c TVE=%"PRIx64,
> +                          addr, is_write ? 'W' : 'R', tve);
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                          " tta=%"PRIx64" lev=%d tts=%d tps=%d",
> +                          tta, lev, tts, tps);
> +        }
> +    }
> +}
> +
> +static IOMMUTLBEntry pnv_phb3_translate_iommu(IOMMUMemoryRegion *iommu,
> +                                              hwaddr addr,
> +                                              IOMMUAccessFlags flag,
> +                                              int iommu_idx)
> +{
> +    PnvPhb3DMASpace *ds = container_of(iommu, PnvPhb3DMASpace, dma_mr);
> +    int tve_sel;
> +    uint64_t tve, cfg;
> +    IOMMUTLBEntry ret = {
> +        .target_as = &address_space_memory,
> +        .iova = addr,
> +        .translated_addr = 0,
> +        .addr_mask = ~(hwaddr)0,
> +        .perm = IOMMU_NONE,
> +    };
> +
> +    /* Resolve PE# */
> +    if (!pnv_phb3_resolve_pe(ds)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> +        return ret;
> +    }
> +
> +    /* Check top bits */
> +    switch (addr >> 60) {
> +    case 00:
> +        /* DMA or 32-bit MSI ? */
> +        cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
> +        if ((cfg & PHB_PHB3C_32BIT_MSI_EN) &&
> +            ((addr & 0xffffffffffff0000ull) == 0xffff0000ull)) {
> +            qemu_log_mask(LOG_GUEST_ERROR, "xlate on 32-bit MSI region");
> +            return ret;
> +        }
> +        /* Choose TVE XXX Use PHB3 Control Register */
> +        tve_sel = (addr >> 59) & 1;
> +        tve = ds->phb->ioda_TVT[ds->pe_num * 2 + tve_sel];
> +        pnv_phb3_translate_tve(ds, addr, flag & IOMMU_WO, tve, &ret);
> +        break;
> +    case 01:
> +        qemu_log_mask(LOG_GUEST_ERROR, "xlate on 64-bit MSI region");
> +        break;
> +    default:
> +        qemu_log_mask(LOG_GUEST_ERROR, "xlate on unsupported address 0x%"PRIx64,
> +                      addr);
> +    }
> +    return ret;
> +}
> +
> +#define TYPE_PNV_PHB3_IOMMU_MEMORY_REGION "pnv-phb3-iommu-memory-region"
> +#define PNV_PHB3_IOMMU_MEMORY_REGION(obj) \
> +    OBJECT_CHECK(IOMMUMemoryRegion, (obj), TYPE_PNV_PHB3_IOMMU_MEMORY_REGION)
> +
> +static void pnv_phb3_iommu_memory_region_class_init(ObjectClass *klass,
> +                                                    void *data)
> +{
> +    IOMMUMemoryRegionClass *imrc = IOMMU_MEMORY_REGION_CLASS(klass);
> +
> +    imrc->translate = pnv_phb3_translate_iommu;
> +}
> +
> +static const TypeInfo pnv_phb3_iommu_memory_region_info = {
> +    .parent = TYPE_IOMMU_MEMORY_REGION,
> +    .name = TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
> +    .class_init = pnv_phb3_iommu_memory_region_class_init,
> +};
> +
> +/*
> + * MSI/MSIX memory region implementation.
> + * The handler handles both MSI and MSIX.
> + */
> +static void pnv_phb3_msi_write(void *opaque, hwaddr addr,
> +                               uint64_t data, unsigned size)
> +{
> +    PnvPhb3DMASpace *ds = opaque;
> +
> +    /* Resolve PE# */
> +    if (!pnv_phb3_resolve_pe(ds)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> +        return;
> +    }
> +
> +    pnv_phb3_msi_send(&ds->phb->msis, addr, data, ds->pe_num);
> +}
> +
> +static const MemoryRegionOps pnv_phb3_msi_ops = {
> +    /* There is no .read as the read result is undefined by PCI spec */

What will qemu do if it hits a NULL read here?  The behaviour may be
undefind, but we'd prefer it didn't crash qemu..

> +    .read = NULL,
> +    .write = pnv_phb3_msi_write,
> +    .endianness = DEVICE_LITTLE_ENDIAN
> +};
> +
> +static AddressSpace *pnv_phb3_dma_iommu(PCIBus *bus, void *opaque, int devfn)
> +{
> +    PnvPHB3 *phb = opaque;
> +    PnvPhb3DMASpace *ds;
> +
> +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> +        if (ds->bus == bus && ds->devfn == devfn) {
> +            break;
> +        }
> +    }
> +
> +    if (ds == NULL) {
> +        ds = g_malloc0(sizeof(PnvPhb3DMASpace));
> +        ds->bus = bus;
> +        ds->devfn = devfn;
> +        ds->pe_num = PHB_INVALID_PE;
> +        ds->phb = phb;
> +        memory_region_init_iommu(&ds->dma_mr, sizeof(ds->dma_mr),
> +                                 TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
> +                                 OBJECT(phb), "phb3_iommu", UINT64_MAX);
> +        address_space_init(&ds->dma_as, MEMORY_REGION(&ds->dma_mr),
> +                           "phb3_iommu");
> +        memory_region_init_io(&ds->msi32_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> +                              ds, "msi32", 0x10000);
> +        memory_region_init_io(&ds->msi64_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> +                              ds, "msi64", 0x100000);
> +        pnv_phb3_update_msi_regions(ds);
> +
> +        QLIST_INSERT_HEAD(&phb->dma_spaces, ds, list);
> +    }
> +    return &ds->dma_as;
> +}
> +
> +static void pnv_phb3_instance_init(Object *obj)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(obj);
> +
> +    /* Create LSI source */
> +    object_initialize(&phb->lsis, sizeof(phb->lsis), TYPE_ICS_SIMPLE);
> +    object_property_add_child(obj, "ics-phb-lsi", OBJECT(&phb->lsis), NULL);
> +
> +    /* Default init ... will be fixed by HW inits */
> +    phb->lsis.offset = 0;
> +
> +    /* Create MSI source */
> +    object_initialize(&phb->msis, sizeof(phb->msis), TYPE_PHB3_MSI);
> +    object_property_add_const_link(OBJECT(&phb->msis), "phb", obj,
> +                                   &error_abort);
> +    object_property_add_child(obj, "ics-phb-msi", OBJECT(&phb->msis), NULL);
> +
> +    /* Create PBCQ */
> +    object_initialize(&phb->pbcq, sizeof(phb->pbcq), TYPE_PNV_PBCQ);
> +    object_property_add_const_link(OBJECT(&phb->pbcq), "phb", obj,
> +                                   &error_abort);
> +    object_property_add_child(obj, "pbcq", OBJECT(&phb->pbcq), NULL);
> +
> +    QLIST_INIT(&phb->dma_spaces);
> +}
> +
> +/*
> + * This could be done under pnv_pbcq_realize
> + */
> +static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
> +{
> +    PCIHostState *pcih = PCI_HOST_BRIDGE(phb);
> +    PCIDevice *brdev;
> +    PCIDevice *pdev;
> +    PCIBus *parent;
> +    uint8_t chassis = phb->chip_id * 4 + phb->phb_id;
> +    uint8_t chassis_nr = 128;
> +
> +    /* Add root complex */
> +    pdev = pci_create(pcih->bus, 0, TYPE_PNV_PHB3_RC);
> +    object_property_add_child(OBJECT(phb), "phb3-rc", OBJECT(pdev), NULL);
> +    qdev_prop_set_uint8(DEVICE(pdev), "chassis", chassis);
> +    qdev_prop_set_uint16(DEVICE(pdev), "slot", 1);
> +    qdev_init_nofail(DEVICE(pdev));
> +
> +    /* Setup bus for that chip */
> +    parent = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
> +
> +    brdev = pci_create(parent, 0, "pci-bridge");

What is this pci bridge representing?  I know PCI-e PHBs typically
have a pseudo P2P bridge right under them, but isn't that represnted
by the root complex above?

> +    object_property_add_child(OBJECT(parent), "pci-bridge", OBJECT(brdev),
> +                              NULL);
> +    qdev_prop_set_uint8(DEVICE(brdev), PCI_BRIDGE_DEV_PROP_CHASSIS_NR,
> +                        chassis_nr);
> +    qdev_init_nofail(DEVICE(brdev));
> +}
> +
> +static void pnv_phb3_realize(DeviceState *dev, Error **errp)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(dev);
> +    PCIHostState *pci = PCI_HOST_BRIDGE(dev);
> +    Object *xics = OBJECT(qdev_get_machine());
> +    Error *local_err = NULL;
> +    int i;
> +
> +    memory_region_init(&phb->pci_mmio, OBJECT(phb), "pci-mmio",
> +                       PCI_MMIO_TOTAL_SIZE);
> +
> +    /* PHB3 doesn't support IO space. However, qemu gets very upset if
> +     * we don't have an IO region to anchor IO BARs onto so we just
> +     * initialize one which we never hook up to anything
> +     */
> +    memory_region_init(&phb->pci_io, OBJECT(phb), "pci-io", 0x10000);
> +
> +    memory_region_init_io(&phb->mr_regs, OBJECT(phb), &pnv_phb3_reg_ops, phb,
> +                          "phb3-regs", 0x1000);
> +
> +    object_property_set_int(OBJECT(&phb->lsis), PNV_PHB3_NUM_LSI, "nr-irqs",
> +                            &error_abort);
> +    object_property_add_const_link(OBJECT(&phb->lsis), "xics", xics,
> +                                   &error_abort);
> +    object_property_set_bool(OBJECT(&phb->lsis), true, "realized", &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    for (i = 0; i < PNV_PHB3_NUM_LSI; i++) {
> +        ics_set_irq_type(&phb->lsis, i, true);
> +    }
> +
> +    object_property_add_const_link(OBJECT(&phb->msis), "xics", xics,
> +                                   &error_abort);
> +    object_property_set_int(OBJECT(&phb->msis), PHB3_MAX_MSI, "nr-irqs",
> +                            &error_abort);
> +    object_property_set_bool(OBJECT(&phb->msis), true, "realized", &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    object_property_set_bool(OBJECT(&phb->pbcq), true, "realized", &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    pci->bus = pci_register_root_bus(dev, "phb3-root-bus",
> +                                pnv_phb3_set_irq, pnv_phb3_map_irq, phb,
> +                                &phb->pci_mmio, &phb->pci_io,
> +                                0, 4, TYPE_PNV_PHB3_ROOT_BUS);
> +    pci_setup_iommu(pci->bus, pnv_phb3_dma_iommu, phb);
> +
> +    /* Setup the PCI busses */
> +    pnv_phb3_pci_create(phb, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +}
> +
> +void pnv_phb3_update_regions(PnvPHB3 *phb)
> +{
> +    PnvPBCQState *pbcq = &phb->pbcq;
> +
> +    /* Unmap first always */
> +    if (phb->regs_mapped) {
> +        memory_region_del_subregion(&pbcq->phbbar, &phb->mr_regs);
> +        phb->regs_mapped = false;
> +    }
> +
> +    /* Map registers if enabled */
> +    if (pbcq->phb_mapped) {
> +        /* XXX We should use the PHB BAR 2 register but we don't ... */
> +        memory_region_add_subregion(&pbcq->phbbar, 0, &phb->mr_regs);
> +        phb->regs_mapped = true;
> +    }
> +
> +    /* Check/update m32 */
> +    if (phb->m32_mapped) {
> +        pnv_phb3_check_m32(phb);
> +    }
> +}
> +
> +static const char *pnv_phb3_root_bus_path(PCIHostState *host_bridge,
> +                                          PCIBus *rootbus)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(host_bridge);
> +
> +    snprintf(phb->bus_path, sizeof(phb->bus_path), "00%02x:%02x",
> +             phb->chip_id, phb->phb_id);
> +    return phb->bus_path;
> +}
> +
> +static void pnv_phb3_get_phb_id(Object *obj, Visitor *v, const char *name,
> +                              void *opaque, Error **errp)
> +{
> +    Property *prop = opaque;
> +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> +
> +    visit_type_uint32(v, name, ptr, errp);
> +}
> +
> +static void pnv_phb3_set_phb_id(Object *obj, Visitor *v, const char *name,
> +                              void *opaque, Error **errp)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(obj);
> +    uint32_t phb_id;
> +    Error *local_err = NULL;
> +
> +    visit_type_uint32(v, name, &phb_id, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    /*
> +     * Limit to a maximum of 6 PHBs per chip
> +     */
> +    if (phb_id >= PNV8_CHIP_PHB3_MAX) {
> +        error_setg(errp, "invalid PHB index: '%d'", phb_id);
> +        return;
> +    }
> +
> +    phb->phb_id = phb_id;
> +}
> +
> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
> +    .name = "irq",
> +    .get = pnv_phb3_get_phb_id,
> +    .set = pnv_phb3_set_phb_id,
> +};

Can't you use a static DeviceProps style property for this, which is a
bit simpler?

> +
> +static void pnv_phb3_get_chip_id(Object *obj, Visitor *v, const char *name,
> +                              void *opaque, Error **errp)
> +{
> +    Property *prop = opaque;
> +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> +
> +    visit_type_uint32(v, name, ptr, errp);
> +}
> +
> +static void pnv_phb3_set_chip_id(Object *obj, Visitor *v, const char *name,
> +                                 void *opaque, Error **errp)
> +{
> +    PnvPHB3 *phb = PNV_PHB3(obj);
> +    uint32_t chip_id;
> +    Error *local_err = NULL;
> +    PnvChip *chip;
> +
> +    visit_type_uint32(v, name, &chip_id, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    /*
> +     * Check that the chip owning the device is valid.
> +     */
> +    chip = pnv_get_chip(PNV_MACHINE(qdev_get_machine()), chip_id);
> +    if (!chip) {
> +        error_setg(errp, "chip id '%d' is invalid", chip_id);
> +        return;
> +    }
> +
> +    phb->chip_id = chip_id;
> +}
> +
> +static const PropertyInfo pnv_phb3_chip_id_propinfo = {
> +    .name = "irq",
> +    .get = pnv_phb3_get_chip_id,
> +    .set = pnv_phb3_set_chip_id,
> +};
> +
> +static Property pnv_phb3_properties[] = {
> +    DEFINE_PROP("phb-id", PnvPHB3, phb_id, pnv_phb3_phb_id_propinfo,
> +                uint32_t),
> +    DEFINE_PROP("chip-id", PnvPHB3, chip_id, pnv_phb3_chip_id_propinfo,
> +                uint32_t),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static void pnv_phb3_class_init(ObjectClass *klass, void *data)
> +{
> +    PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_CLASS(klass);
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +
> +    hc->root_bus_path = pnv_phb3_root_bus_path;
> +    dc->realize = pnv_phb3_realize;
> +    dc->props = pnv_phb3_properties;
> +    set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
> +}
> +
> +static const TypeInfo pnv_phb3_type_info = {
> +    .name          = TYPE_PNV_PHB3,
> +    .parent        = TYPE_PCI_HOST_BRIDGE,
> +    .instance_size = sizeof(PnvPHB3),
> +    .class_init    = pnv_phb3_class_init,
> +    .instance_init = pnv_phb3_instance_init,
> +};
> +
> +static void pnv_phb3_root_bus_class_init(ObjectClass *klass, void *data)
> +{
> +    BusClass *k = BUS_CLASS(klass);
> +
> +    k->max_dev = 1;
> +}
> +
> +static const TypeInfo pnv_phb3_root_bus_info = {
> +    .name = TYPE_PNV_PHB3_ROOT_BUS,
> +    .parent = TYPE_PCIE_BUS,
> +    .class_init = pnv_phb3_root_bus_class_init,
> +};
> +
> +static void pnv_phb3_rc_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
> +    PCIERootPortClass *rpc = PCIE_ROOT_PORT_CLASS(klass);
> +
> +    dc->desc     = "IBM PHB3 PCIE Root Port";
> +
> +    k->vendor_id = PCI_VENDOR_ID_IBM;
> +    k->device_id = 0x03dc;
> +    k->revision  = 0;
> +    rpc->exp_offset = 0x48;
> +    rpc->aer_offset = 0x100;
> +}
> +
> +static const TypeInfo pnv_phb3_rc_info = {
> +    .name          = TYPE_PNV_PHB3_RC,
> +    .parent        = TYPE_PCIE_ROOT_PORT,
> +    .class_init    = pnv_phb3_rc_class_init,
> +};
> +
> +static void pnv_phb3_register_types(void)
> +{
> +    type_register_static(&pnv_phb3_rc_info);
> +    type_register_static(&pnv_phb3_type_info);
> +    type_register_static(&pnv_phb3_root_bus_info);
> +    type_register_static(&pnv_phb3_iommu_memory_region_info);
> +}
> +
> +type_init(pnv_phb3_register_types)
> diff --git a/hw/pci-host/pnv_phb3_msi.c b/hw/pci-host/pnv_phb3_msi.c
> new file mode 100644
> index 000000000000..36927da25b61
> --- /dev/null
> +++ b/hw/pci-host/pnv_phb3_msi.c
> @@ -0,0 +1,316 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qemu/log.h"
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "target/ppc/cpu.h"
> +#include "hw/pci-host/pnv_phb3_regs.h"
> +#include "hw/pci-host/pnv_phb3.h"
> +#include "hw/ppc/pnv.h"
> +#include "hw/pci/msi.h"
> +
> +static uint64_t phb3_msi_ive_addr(PnvPHB3 *phb, int srcno)
> +{
> +    uint64_t ivtbar = phb->regs[PHB_IVT_BAR >> 3];
> +    uint64_t phbctl = phb->regs[PHB_CONTROL >> 3];
> +
> +    if (!(ivtbar & PHB_IVT_BAR_ENABLE)) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Failed access to disable IVT BAR !");
> +        return 0;
> +    }
> +
> +    if (srcno >= (ivtbar & PHB_IVT_LENGTH_MASK)) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "MSI out of bounds (%d vs  0x%"PRIx64")",
> +                      srcno, ivtbar & PHB_IVT_LENGTH_MASK);
> +        return 0;
> +    }
> +
> +    ivtbar &= PHB_IVT_BASE_ADDRESS_MASK;
> +
> +    if (phbctl & PHB_CTRL_IVE_128_BYTES) {
> +        return ivtbar + 128 * srcno;
> +    } else {
> +        return ivtbar + 16 * srcno;
> +    }
> +}
> +
> +static bool phb3_msi_read_ive(PnvPHB3 *phb, int srcno, uint64_t *out_ive)
> +{
> +    uint64_t ive_addr, ive;
> +
> +    ive_addr = phb3_msi_ive_addr(phb, srcno);
> +    if (!ive_addr) {
> +        return false;
> +    }
> +
> +    if (dma_memory_read(&address_space_memory, ive_addr, &ive, sizeof(ive))) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read IVE at 0x%" PRIx64,
> +                      ive_addr);
> +        return false;
> +    }
> +    *out_ive = be64_to_cpu(ive);
> +
> +    return true;
> +}
> +
> +static void phb3_msi_set_p(Phb3MsiState *msi, int srcno, uint8_t gen)
> +{
> +    uint64_t ive_addr;
> +    uint8_t p = 0x01 | (gen << 1);
> +
> +    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
> +    if (!ive_addr) {
> +        return;
> +    }
> +
> +    if (dma_memory_write(&address_space_memory, ive_addr + 4, &p, 1)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to write IVE (set P) at 0x%" PRIx64, ive_addr);
> +    }
> +}
> +
> +static void phb3_msi_set_q(Phb3MsiState *msi, int srcno)
> +{
> +    uint64_t ive_addr;
> +    uint8_t q = 0x01;
> +
> +    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
> +    if (!ive_addr) {
> +        return;
> +    }
> +
> +    if (dma_memory_write(&address_space_memory, ive_addr + 5, &q, 1)) {
> +        qemu_log_mask(LOG_GUEST_ERROR,
> +                      "Failed to write IVE (set Q) at 0x%" PRIx64, ive_addr);
> +    }
> +}
> +
> +static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
> +{
> +    ICSState *ics = ICS_BASE(msi);
> +    uint64_t ive;
> +    uint64_t server, prio, pq, gen;
> +
> +    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
> +        return;
> +    }
> +
> +    server = GETFIELD(IODA2_IVT_SERVER, ive);
> +    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
> +    pq = GETFIELD(IODA2_IVT_Q, ive);
> +    if (!ignore_p) {
> +        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
> +    }
> +    gen = GETFIELD(IODA2_IVT_GEN, ive);
> +
> +    /*
> +     * The low order 2 bits are the link pointer (Type II interrupts).
> +     * Shift back to get a valid IRQ server.
> +     */
> +    server >>= 2;
> +
> +    switch (pq) {
> +    case 0: /* 00 */
> +        if (prio == 0xff) {
> +            /* Masked, set Q */
> +            phb3_msi_set_q(msi, srcno);
> +        } else {
> +            /* Enabled, set P and send */
> +            phb3_msi_set_p(msi, srcno, gen);
> +            icp_irq(ics, server, srcno + ics->offset, prio);

Can't you just pulse the right qirq here, which will use the core ICS
logic to propagate to the ICP?

> +        }
> +        break;
> +    case 2: /* 10 */
> +        /* Already pending, set Q */
> +        phb3_msi_set_q(msi, srcno);
> +        break;
> +    case 1: /* 01 */
> +    case 3: /* 11 */
> +    default:
> +        /* Just drop stuff if Q already set */
> +        break;
> +    }
> +}
> +
> +static void phb3_msi_set_irq(void *opaque, int srcno, int val)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(opaque);
> +
> +    if (val) {
> +        phb3_msi_try_send(msi, srcno, false);
> +    }
> +}
> +
> +
> +void pnv_phb3_msi_send(Phb3MsiState *msi, uint64_t addr, uint16_t data,
> +                       int32_t dev_pe)
> +{
> +    ICSState *ics = ICS_BASE(msi);
> +    uint64_t ive;
> +    uint16_t pe;
> +    uint32_t src = ((addr >> 4) & 0xffff) | (data & 0x1f);
> +
> +    if (src >= ics->nr_irqs) {
> +        qemu_log_mask(LOG_GUEST_ERROR, "MSI %d out of bounds", src);
> +        return;
> +    }
> +    if (dev_pe >= 0) {
> +        if (!phb3_msi_read_ive(msi->phb, src, &ive)) {
> +            return;
> +        }
> +        pe = GETFIELD(IODA2_IVT_PE, ive);
> +        if (pe != dev_pe) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                          "MSI %d send by PE#%d but assigned to PE#%d",
> +                          src, dev_pe, pe);
> +            return;
> +        }
> +    }
> +    qemu_irq_pulse(ics->qirqs[src]);
> +}
> +
> +void pnv_phb3_msi_ffi(Phb3MsiState *msi, uint64_t val)
> +{
> +    /* Emit interrupt */
> +    pnv_phb3_msi_send(msi, val, 0, -1);
> +
> +    /* Clear FFI lock */
> +    msi->phb->regs[PHB_FFI_LOCK >> 3] = 0;
> +}
> +
> +static void phb3_msi_reject(ICSState *ics, uint32_t nr)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(ics);
> +    unsigned int srcno = nr - ics->offset;
> +    unsigned int idx = srcno >> 6;
> +    unsigned int bit = 1ull << (srcno & 0x3f);
> +
> +    assert(srcno < PHB3_MAX_MSI);
> +
> +    msi->rba[idx] |= bit;
> +    msi->rba_sum |= (1u << idx);
> +}
> +
> +static void phb3_msi_resend(ICSState *ics)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(ics);
> +    unsigned int i, j;
> +
> +    if (msi->rba_sum == 0) {
> +        return;
> +    }
> +
> +    for (i = 0; i < 32; i++) {
> +        if ((msi->rba_sum & (1u << i)) == 0) {
> +            continue;
> +        }
> +        msi->rba_sum &= ~(1u << i);
> +        for (j = 0; j < 64; j++) {
> +            if ((msi->rba[i] & (1ull << j)) == 0) {
> +                continue;
> +            }
> +            msi->rba[i] &= ~(1u << j);
> +            phb3_msi_try_send(msi, i * 64 + j, true);
> +        }
> +    }
> +}
> +
> +static void phb3_msi_reset(DeviceState *dev)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(dev);
> +    ICSStateClass *icsc = ICS_BASE_GET_CLASS(dev);
> +
> +    icsc->parent_reset(dev);
> +
> +    memset(msi->rba, 0, sizeof(msi->rba));
> +    msi->rba_sum = 0;
> +}
> +
> +static void phb3_msi_reset_handler(void *dev)
> +{
> +    phb3_msi_reset(dev);
> +}
> +
> +void pnv_phb3_msi_update_config(Phb3MsiState *msi, uint32_t base,
> +                                uint32_t count)
> +{
> +    ICSState *ics = ICS_BASE(msi);
> +
> +    if (count > PHB3_MAX_MSI) {
> +        count = PHB3_MAX_MSI;
> +    }
> +    ics->nr_irqs = count;
> +    ics->offset = base;
> +}
> +
> +static void phb3_msi_realize(DeviceState *dev, Error **errp)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(dev);
> +    ICSState *ics = ICS_BASE(msi);
> +    ICSStateClass *icsc = ICS_BASE_GET_CLASS(ics);
> +    Object *obj;
> +    Error *local_err = NULL;
> +
> +    icsc->parent_realize(dev, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
> +    if (!obj) {
> +        error_propagate(errp, local_err);
> +        error_prepend(errp, "required link 'phb' not found: ");
> +        return;
> +    }
> +    msi->phb = PNV_PHB3(obj);
> +
> +    ics->qirqs = qemu_allocate_irqs(phb3_msi_set_irq, msi, PHB3_MAX_MSI);
> +
> +    qemu_register_reset(phb3_msi_reset_handler, dev);
> +}
> +
> +static void phb3_msi_instance_init(Object *obj)
> +{
> +    ICSState *ics = ICS_BASE(obj);
> +
> +    /* Will be overriden later */
> +    ics->offset = 0;
> +}
> +
> +static void phb3_msi_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    ICSStateClass *isc = ICS_BASE_CLASS(klass);
> +
> +    device_class_set_parent_realize(dc, phb3_msi_realize,
> +                                    &isc->parent_realize);
> +    device_class_set_parent_reset(dc, phb3_msi_reset,
> +                                  &isc->parent_reset);
> +
> +    isc->reject = phb3_msi_reject;
> +    isc->resend = phb3_msi_resend;
> +}
> +
> +static const TypeInfo phb3_msi_info = {
> +    .name = TYPE_PHB3_MSI,
> +    .parent = TYPE_ICS_BASE,
> +    .instance_size = sizeof(Phb3MsiState),
> +    .class_init = phb3_msi_class_init,
> +    .class_size = sizeof(ICSStateClass),
> +    .instance_init = phb3_msi_instance_init,
> +};
> +
> +static void pnv_phb3_msi_register_types(void)
> +{
> +    type_register_static(&phb3_msi_info);
> +}
> +
> +type_init(pnv_phb3_msi_register_types)
> diff --git a/hw/pci-host/pnv_phb3_pbcq.c b/hw/pci-host/pnv_phb3_pbcq.c
> new file mode 100644
> index 000000000000..34df3ac684e4
> --- /dev/null
> +++ b/hw/pci-host/pnv_phb3_pbcq.c
> @@ -0,0 +1,347 @@
> +/*
> + * QEMU PowerPC PowerNV PHB3 model
> + *
> + * Copyright (c) 2014-2018, IBM Corporation.
> + *
> + * This code is licensed under the GPL version 2 or later. See the
> + * COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "qemu/log.h"
> +#include "target/ppc/cpu.h"
> +#include "hw/ppc/fdt.h"
> +#include "hw/pci-host/pnv_phb3_regs.h"
> +#include "hw/pci-host/pnv_phb3.h"
> +#include "hw/ppc/pnv_xscom.h"
> +#include "hw/pci/pci_bridge.h"
> +#include "hw/pci/pci_bus.h"
> +
> +#include <libfdt.h>
> +
> +static uint64_t pnv_pbcq_nest_xscom_read(void *opaque, hwaddr addr,
> +                                         unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t offset = addr >> 3;
> +
> +    return pbcq->nest_regs[offset];
> +}
> +
> +static uint64_t pnv_pbcq_pci_xscom_read(void *opaque, hwaddr addr,
> +                                        unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t offset = addr >> 3;
> +
> +    return pbcq->pci_regs[offset];
> +}
> +
> +static uint64_t pnv_pbcq_spci_xscom_read(void *opaque, hwaddr addr,
> +                                         unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t offset = addr >> 3;
> +
> +    if (offset == PBCQ_SPCI_ASB_DATA) {
> +        return pnv_phb3_reg_read(pbcq->phb,
> +                                 pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR], 8);
> +    }
> +    return pbcq->spci_regs[offset];
> +}
> +
> +static void pnv_pbcq_update_map(PnvPBCQState *pbcq)
> +{
> +    uint64_t bar_en = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
> +    uint64_t bar, mask, size;
> +
> +    /*
> +     * NOTE: This will really not work well if those are remapped
> +     * after the PHB has created its sub regions. We could do better
> +     * if we had a way to resize regions but we don't really care
> +     * that much in practice as the stuff below really only happens
> +     * once early during boot
> +     */
> +
> +    /* Handle unmaps */
> +    if (pbcq->mmio0_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
> +        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar0);
> +        pbcq->mmio0_mapped = false;
> +    }
> +    if (pbcq->mmio1_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
> +        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar1);
> +        pbcq->mmio1_mapped = false;
> +    }
> +    if (pbcq->phb_mapped && !(bar_en & PBCQ_NEST_BAR_EN_PHB)) {
> +        memory_region_del_subregion(get_system_memory(), &pbcq->phbbar);
> +        pbcq->phb_mapped = false;
> +    }
> +
> +    /* Update PHB */
> +    pnv_phb3_update_regions(pbcq->phb);
> +
> +    /* Handle maps */
> +    if (!pbcq->mmio0_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
> +        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] >> 14;
> +        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0];
> +        size = ((~mask) >> 14) + 1;
> +        memory_region_init(&pbcq->mmbar0, OBJECT(pbcq), "pbcq-mmio0", size);
> +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar0);
> +        pbcq->mmio0_mapped = true;
> +        pbcq->mmio0_base = bar;
> +        pbcq->mmio0_size = size;
> +    }
> +    if (!pbcq->mmio1_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
> +        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] >> 14;
> +        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1];
> +        size = ((~mask) >> 14) + 1;
> +        memory_region_init(&pbcq->mmbar1, OBJECT(pbcq), "pbcq-mmio1", size);
> +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar1);
> +        pbcq->mmio1_mapped = true;
> +        pbcq->mmio1_base = bar;
> +        pbcq->mmio1_size = size;
> +    }
> +    if (!pbcq->phb_mapped && (bar_en & PBCQ_NEST_BAR_EN_PHB)) {
> +        bar = pbcq->nest_regs[PBCQ_NEST_PHB_BAR] >> 14;
> +        size = 0x1000;
> +        memory_region_init(&pbcq->phbbar, OBJECT(pbcq), "pbcq-phb", size);
> +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->phbbar);
> +        pbcq->phb_mapped = true;
> +    }
> +
> +    /* Update PHB */
> +    pnv_phb3_update_regions(pbcq->phb);
> +}
> +
> +static void pnv_pbcq_nest_xscom_write(void *opaque, hwaddr addr,
> +                                uint64_t val, unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t reg = addr >> 3;
> +
> +    switch (reg) {
> +    case PBCQ_NEST_MMIO_BAR0:
> +    case PBCQ_NEST_MMIO_BAR1:
> +    case PBCQ_NEST_MMIO_MASK0:
> +    case PBCQ_NEST_MMIO_MASK1:
> +        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] &
> +            (PBCQ_NEST_BAR_EN_MMIO0 |
> +             PBCQ_NEST_BAR_EN_MMIO1)) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                           "PHB3: Changing enabled BAR unsupported\n");
> +        }
> +        pbcq->nest_regs[reg] = val & 0xffffffffc0000000ull;
> +        break;
> +    case PBCQ_NEST_PHB_BAR:
> +        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] & PBCQ_NEST_BAR_EN_PHB) {
> +            qemu_log_mask(LOG_GUEST_ERROR,
> +                           "PHB3: Changing enabled BAR unsupported\n");
> +        }
> +        pbcq->nest_regs[reg] = val & 0xfffffffffc000000ull;
> +        break;
> +    case PBCQ_NEST_BAR_EN:
> +        pbcq->nest_regs[reg] = val & 0xf800000000000000ull;
> +        pnv_pbcq_update_map(pbcq);
> +        pnv_phb3_remap_irqs(pbcq->phb);
> +        break;
> +    case PBCQ_NEST_IRSN_COMPARE:
> +    case PBCQ_NEST_IRSN_MASK:
> +        pbcq->nest_regs[reg] = val & PBCQ_NEST_IRSN_COMP;
> +        pnv_phb3_remap_irqs(pbcq->phb);
> +        break;
> +    case PBCQ_NEST_LSI_SRC_ID:
> +        pbcq->nest_regs[reg] = val & PBCQ_NEST_LSI_SRC;
> +        pnv_phb3_remap_irqs(pbcq->phb);
> +        break;
> +    }
> +
> +    /* XXX Don't error out on other regs for now ... */
> +}
> +
> +static void pnv_pbcq_pci_xscom_write(void *opaque, hwaddr addr,
> +                                     uint64_t val, unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t reg = addr >> 3;
> +
> +    switch (reg) {
> +    case PBCQ_PCI_BAR2:
> +        pbcq->pci_regs[reg] = val & 0xfffffffffc000000ull;
> +        pnv_pbcq_update_map(pbcq);
> +        break;
> +    }
> +
> +    /* XXX Don't error out on other regs for now ... */
> +}
> +
> +static void pnv_pbcq_spci_xscom_write(void *opaque, hwaddr addr,
> +                                uint64_t val, unsigned size)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> +    uint32_t reg = addr >> 3;
> +
> +    switch (reg) {
> +    case PBCQ_SPCI_ASB_ADDR:
> +        pbcq->spci_regs[reg] = val & 0xfff;
> +        break;
> +    case PBCQ_SPCI_ASB_STATUS:
> +        pbcq->spci_regs[reg] &= ~val;
> +        break;
> +    case PBCQ_SPCI_ASB_DATA:
> +        pnv_phb3_reg_write(pbcq->phb, pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR],
> +                           val, 8);
> +        break;
> +    case PBCQ_SPCI_AIB_CAPP_EN:
> +    case PBCQ_SPCI_CAPP_SEC_TMR:
> +        break;
> +    }
> +
> +    /* XXX Don't error out on other regs for now ... */
> +}
> +
> +static const MemoryRegionOps pnv_pbcq_nest_xscom_ops = {
> +    .read = pnv_pbcq_nest_xscom_read,
> +    .write = pnv_pbcq_nest_xscom_write,
> +    .valid.min_access_size = 8,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 8,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static const MemoryRegionOps pnv_pbcq_pci_xscom_ops = {
> +    .read = pnv_pbcq_pci_xscom_read,
> +    .write = pnv_pbcq_pci_xscom_write,
> +    .valid.min_access_size = 8,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 8,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static const MemoryRegionOps pnv_pbcq_spci_xscom_ops = {
> +    .read = pnv_pbcq_spci_xscom_read,
> +    .write = pnv_pbcq_spci_xscom_write,
> +    .valid.min_access_size = 8,
> +    .valid.max_access_size = 8,
> +    .impl.min_access_size = 8,
> +    .impl.max_access_size = 8,
> +    .endianness = DEVICE_BIG_ENDIAN,
> +};
> +
> +static void pnv_pbcq_default_bars(PnvPBCQState *pbcq)
> +{
> +    uint64_t mm0, mm1, reg;
> +    PnvPHB3 *phb = pbcq->phb;
> +
> +    mm0 = 0x3d00000000000ull + 0x4000000000ull * phb->chip_id +
> +            0x1000000000ull * phb->phb_id;
> +    mm1 = 0x3ff8000000000ull + 0x0200000000ull * phb->chip_id +
> +            0x0080000000ull * phb->phb_id;
> +    reg = 0x3fffe40000000ull + 0x0000400000ull * phb->chip_id +
> +            0x0000100000ull * phb->phb_id;
> +
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] = mm0 << 14;
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] = mm1 << 14;
> +    pbcq->nest_regs[PBCQ_NEST_PHB_BAR] = reg << 14;
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0] = 0x3fff000000000ull << 14;
> +    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1] = 0x3ffff80000000ull << 14;
> +    pbcq->pci_regs[PBCQ_PCI_BAR2] = reg << 14;
> +}
> +
> +static void pnv_pbcq_realize(DeviceState *dev, Error **errp)
> +{
> +    PnvPBCQState *pbcq = PNV_PBCQ(dev);
> +    PnvPHB3 *phb;
> +    Object *obj;
> +    Error *local_err = NULL;
> +    char name[32];
> +
> +    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
> +    if (!obj) {
> +        error_propagate(errp, local_err);
> +        error_prepend(errp, "required link 'phb' not found: ");
> +        return;
> +    }
> +    phb = pbcq->phb = PNV_PHB3(obj);
> +
> +    /* XXX Fix OPAL to do that: establish default BAR values */
> +    pnv_pbcq_default_bars(pbcq);
> +
> +    /* Initialize the XSCOM region for the PBCQ registers */
> +    snprintf(name, sizeof(name), "xscom-pbcq-nest-%d.%d",
> +             phb->chip_id, phb->phb_id);
> +    pnv_xscom_region_init(&pbcq->xscom_nest_regs, OBJECT(dev),
> +                          &pnv_pbcq_nest_xscom_ops, pbcq, name,
> +                          PNV_XSCOM_PBCQ_NEST_SIZE);
> +    snprintf(name, sizeof(name), "xscom-pbcq-pci-%d.%d",
> +             phb->chip_id, phb->phb_id);
> +    pnv_xscom_region_init(&pbcq->xscom_pci_regs, OBJECT(dev),
> +                          &pnv_pbcq_pci_xscom_ops, pbcq, name,
> +                          PNV_XSCOM_PBCQ_PCI_SIZE);
> +    snprintf(name, sizeof(name), "xscom-pbcq-spci-%d.%d",
> +             phb->chip_id, phb->phb_id);
> +    pnv_xscom_region_init(&pbcq->xscom_spci_regs, OBJECT(dev),
> +                          &pnv_pbcq_spci_xscom_ops, pbcq, name,
> +                          PNV_XSCOM_PBCQ_SPCI_SIZE);
> +}
> +
> +static int pnv_pbcq_dt_xscom(PnvXScomInterface *dev, void *fdt,
> +                             int xscom_offset)
> +{
> +    const char compat[] = "ibm,power8-pbcq";
> +    PnvPHB3 *phb = PNV_PBCQ(dev)->phb;
> +    char *name;
> +    int offset;
> +    uint32_t lpc_pcba = PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id;
> +    uint32_t reg[] = {
> +        cpu_to_be32(lpc_pcba),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_NEST_SIZE),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_SIZE),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id),
> +        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_SIZE)
> +    };
> +
> +    name = g_strdup_printf("pbcq@%x", lpc_pcba);
> +    offset = fdt_add_subnode(fdt, xscom_offset, name);
> +    _FDT(offset);
> +    g_free(name);
> +
> +    _FDT((fdt_setprop(fdt, offset, "reg", reg, sizeof(reg))));
> +
> +    _FDT((fdt_setprop_cell(fdt, offset, "ibm,phb-index", phb->phb_id)));
> +    _FDT((fdt_setprop_cell(fdt, offset, "ibm,chip-id", phb->chip_id)));
> +    _FDT((fdt_setprop(fdt, offset, "compatible", compat,
> +                      sizeof(compat))));
> +    return 0;
> +}
> +
> +static void pnv_pbcq_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    PnvXScomInterfaceClass *xdc = PNV_XSCOM_INTERFACE_CLASS(klass);
> +
> +    xdc->dt_xscom = pnv_pbcq_dt_xscom;
> +
> +    dc->realize = pnv_pbcq_realize;
> +}
> +
> +static const TypeInfo pnv_pbcq_type_info = {
> +    .name          = TYPE_PNV_PBCQ,
> +    .parent        = TYPE_DEVICE,
> +    .instance_size = sizeof(PnvPBCQState),
> +    .class_init    = pnv_pbcq_class_init,
> +    .interfaces    = (InterfaceInfo[]) {
> +        { TYPE_PNV_XSCOM_INTERFACE },
> +        { }
> +    }
> +};
> +
> +static void pnv_pbcq_register_types(void)
> +{
> +    type_register_static(&pnv_pbcq_type_info);
> +}
> +
> +type_init(pnv_pbcq_register_types)
> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> index 7401ffe5b01c..77e92d5e2e66 100644
> --- a/hw/ppc/pnv.c
> +++ b/hw/ppc/pnv.c
> @@ -44,6 +44,7 @@
>  #include "hw/isa/isa.h"
>  #include "hw/char/serial.h"
>  #include "hw/timer/mc146818rtc.h"
> +#include "hw/pci/msi.h"
>  
>  #include <libfdt.h>
>  
> @@ -555,6 +556,9 @@ static void pnv_init(MachineState *machine)
>      int i;
>      char *chip_typename;
>  
> +    /* MSIs are supported on this platform */
> +    msi_nonbroken = true;
> +
>      /* allocate RAM */
>      if (machine->ram_size < (1 * G_BYTE)) {
>          warn_report("skiboot may not work with < 1GB of RAM");
> @@ -725,6 +729,8 @@ static Object *pnv_chip_power9_intc_create(PnvChip *chip, Object *child,
>  static void pnv_chip_power8_instance_init(Object *obj)
>  {
>      Pnv8Chip *chip8 = PNV8_CHIP(obj);
> +    PnvChipClass *pcc = PNV_CHIP_GET_CLASS(obj);
> +    int i;
>  
>      object_initialize(&chip8->psi, sizeof(chip8->psi), TYPE_PNV_PSI);
>      object_property_add_child(obj, "psi", OBJECT(&chip8->psi), NULL);
> @@ -740,6 +746,12 @@ static void pnv_chip_power8_instance_init(Object *obj)
>      object_property_add_child(obj, "occ", OBJECT(&chip8->occ), NULL);
>      object_property_add_const_link(OBJECT(&chip8->occ), "psi",
>                                     OBJECT(&chip8->psi), &error_abort);
> +
> +    for (i = 0; i < pcc->num_phbs; i++) {
> +        object_initialize(&chip8->phbs[i], sizeof(chip8->phbs[i]),
> +                          TYPE_PNV_PHB3);
> +        object_property_add_child(obj, "phb[*]", OBJECT(&chip8->phbs[i]), NULL);
> +    }
>  }
>  
>  static void pnv_chip_icp_realize(Pnv8Chip *chip8, Error **errp)
> @@ -780,6 +792,7 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
>      PnvChip *chip = PNV_CHIP(dev);
>      Pnv8Chip *chip8 = PNV8_CHIP(dev);
>      Error *local_err = NULL;
> +    int i;
>  
>      pcc->parent_realize(dev, &local_err);
>      if (local_err) {
> @@ -817,6 +830,33 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
>          return;
>      }
>      pnv_xscom_add_subregion(chip, PNV_XSCOM_OCC_BASE, &chip8->occ.xscom_regs);
> +
> +    /* Create the PHB3 controllers */
> +    for (i = 0; i < pcc->num_phbs; i++) {
> +        PnvPHB3 *phb = &chip8->phbs[i];
> +        PnvPBCQState *pbcq = &phb->pbcq;
> +
> +        object_property_set_int(OBJECT(phb), i, "phb-id", &error_fatal);
> +        object_property_set_int(OBJECT(phb), chip->chip_id, "chip-id",
> +                                &error_fatal);
> +        object_property_set_bool(OBJECT(phb), true, "realized", &local_err);
> +        if (local_err) {
> +            error_propagate(errp, local_err);
> +            return;
> +        }
> +        qdev_set_parent_bus(DEVICE(phb), sysbus_get_default());
> +
> +        /* Populate the XSCOM address space. */
> +        pnv_xscom_add_subregion(chip,
> +                                PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
> +                                &pbcq->xscom_nest_regs);
> +        pnv_xscom_add_subregion(chip,
> +                                PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
> +                                &pbcq->xscom_pci_regs);
> +        pnv_xscom_add_subregion(chip,
> +                                PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
> +                                &pbcq->xscom_spci_regs);
> +    }
>  }
>  
>  static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
> @@ -827,6 +867,7 @@ static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER8E;
>      k->chip_cfam_id = 0x221ef04980000000ull;  /* P8 Murano DD2.1 */
>      k->cores_mask = POWER8E_CORE_MASK;
> +    k->num_phbs = 2;
>      k->core_pir = pnv_chip_core_pir_p8;
>      k->intc_create = pnv_chip_power8_intc_create;
>      k->isa_create = pnv_chip_power8_isa_create;
> @@ -845,6 +886,7 @@ static void pnv_chip_power8_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER8;
>      k->chip_cfam_id = 0x220ea04980000000ull; /* P8 Venice DD2.0 */
>      k->cores_mask = POWER8_CORE_MASK;
> +    k->num_phbs = 3;
>      k->core_pir = pnv_chip_core_pir_p8;
>      k->intc_create = pnv_chip_power8_intc_create;
>      k->isa_create = pnv_chip_power8_isa_create;
> @@ -863,6 +905,7 @@ static void pnv_chip_power8nvl_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER8NVL;
>      k->chip_cfam_id = 0x120d304980000000ull;  /* P8 Naples DD1.0 */
>      k->cores_mask = POWER8_CORE_MASK;
> +    k->num_phbs = 4;
>      k->core_pir = pnv_chip_core_pir_p8;
>      k->intc_create = pnv_chip_power8_intc_create;
>      k->isa_create = pnv_chip_power8nvl_isa_create;
> @@ -897,6 +940,7 @@ static void pnv_chip_power9_class_init(ObjectClass *klass, void *data)
>      k->chip_type = PNV_CHIP_POWER9;
>      k->chip_cfam_id = 0x220d104900008000ull; /* P9 Nimbus DD2.0 */
>      k->cores_mask = POWER9_CORE_MASK;
> +    k->num_phbs = 0;
>      k->core_pir = pnv_chip_core_pir_p9;
>      k->intc_create = pnv_chip_power9_intc_create;
>      k->isa_create = pnv_chip_power9_isa_create;
> @@ -1047,14 +1091,24 @@ static void pnv_chip_class_init(ObjectClass *klass, void *data)
>  static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
>  {
>      PnvMachineState *pnv = PNV_MACHINE(xi);
> -    int i;
> +    int i, j;
>  
>      for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
>          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
>  
>          if (ics_valid_irq(&chip8->psi.ics, irq)) {
>              return &chip8->psi.ics;
>          }
> +        for (j = 0; j < pcc->num_phbs; j++) {
> +            if (ics_valid_irq(&chip8->phbs[j].lsis, irq)) {
> +                return &chip8->phbs[j].lsis;
> +            }
> +            if (ics_valid_irq(ICS_BASE(&chip8->phbs[j].msis), irq)) {
> +                return ICS_BASE(&chip8->phbs[j].msis);
> +            }
> +        }
>      }
>      return NULL;
>  }
> @@ -1062,11 +1116,18 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
>  static void pnv_ics_resend(XICSFabric *xi)
>  {
>      PnvMachineState *pnv = PNV_MACHINE(xi);
> -    int i;
> +    int i, j;
>  
>      for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
>          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
> +
>          ics_resend(&chip8->psi.ics);
> +        for (j = 0; j < pcc->num_phbs; j++) {
> +            ics_resend(&chip8->phbs[j].lsis);
> +            ics_resend(ICS_BASE(&chip8->phbs[j].msis));
> +        }
>      }
>  }
>  
> @@ -1097,7 +1158,7 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
>                                 Monitor *mon)
>  {
>      PnvMachineState *pnv = PNV_MACHINE(obj);
> -    int i;
> +    int i, j;
>      CPUState *cs;
>  
>      CPU_FOREACH(cs) {
> @@ -1107,8 +1168,16 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
>      }
>  
>      for (i = 0; i < pnv->num_chips; i++) {
> +        PnvChip *chip = pnv->chips[i];
>          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
> +
>          ics_pic_print_info(&chip8->psi.ics, mon);
> +
> +        for (j = 0; j < pcc->num_phbs; j++) {
> +            ics_pic_print_info(&chip8->phbs[j].lsis, mon);
> +            ics_pic_print_info(ICS_BASE(&chip8->phbs[j].msis), mon);
> +        }
>      }
>  }
>  
> diff --git a/hw/ppc/pnv_xscom.c b/hw/ppc/pnv_xscom.c
> index 46fae41f32b0..83f05d06053f 100644
> --- a/hw/ppc/pnv_xscom.c
> +++ b/hw/ppc/pnv_xscom.c
> @@ -252,7 +252,11 @@ int pnv_dt_xscom(PnvChip *chip, void *fdt, int root_offset)
>      args.fdt = fdt;
>      args.xscom_offset = xscom_offset;
>  
> -    object_child_foreach(OBJECT(chip), xscom_dt_child, &args);
> +    /* Some PnvXScomInterface objects lie a bit deeper (PnvPBCQState)
> +     * than the first layer, so we need to loop on the whole object
> +     * hierarchy to catch them
> +     */
> +    object_child_foreach_recursive(OBJECT(chip), xscom_dt_child, &args);
>      return 0;
>  }
>  
> diff --git a/hw/pci-host/Makefile.objs b/hw/pci-host/Makefile.objs
> index 6d6597c06563..1b5b782bd986 100644
> --- a/hw/pci-host/Makefile.objs
> +++ b/hw/pci-host/Makefile.objs
> @@ -19,3 +19,4 @@ common-obj-$(CONFIG_PCI_GENERIC) += gpex.o
>  common-obj-$(CONFIG_PCI_XILINX) += xilinx-pcie.o
>  
>  common-obj-$(CONFIG_PCI_DESIGNWARE) += designware.o
> +obj-$(CONFIG_POWERNV) += pnv_phb3.o pnv_phb3_pbcq.o pnv_phb3_msi.o

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-09  7:22   ` Cédric Le Goater
@ 2018-07-18  6:13     ` David Gibson
  0 siblings, 0 replies; 29+ messages in thread
From: David Gibson @ 2018-07-18  6:13 UTC (permalink / raw)
  To: Cédric Le Goater
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

[-- Attachment #1: Type: text/plain, Size: 113294 bytes --]

On Mon, Jul 09, 2018 at 09:22:59AM +0200, Cédric Le Goater wrote:
> On 06/28/2018 10:36 AM, Cédric Le Goater wrote:
> > From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > 
> > This is a model of the PCIe Host Bridge (PHB3) found on a Power8
> > processor. It includes the PowerBus logic interface (PBCQ), IOMMU
> > support, PCIe root complex, the XICS MSI and LSI interrupt sources.
> > 
> > The Power8 processor comes in different flavors: Venice, Murano,
> > Naple, each having a different number of PHBs. All are initialized but
> > the machine only needs one PCI bus to plug a couple of devices (net,
> > storage, usbs). The other busses are provided to test complex
> > configurations. Some platforms, like the Firestone, can also couple
> > PHBs on the first chip to provide more bandwidth but this is too
> > specific to model in QEMU.
> > 
> > No default device layout is provided and PCI devices can be added on
> > any of the available PCI busses (pci.0..2 on a Power8 chip) using
> > command line options such as :
> > 
> >   -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.0,addr=0x2
> >   -netdev bridge,id=net0,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=hostnet0
> > 
> >   -device megasas,id=scsi0,bus=pci.0,addr=0x1
> >   -drive file=$disk,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none
> >   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2
> > 
> > Multi chip is supported, each chip adding its set of PHB3 controllers
> > and its PCI busses. The model doesn't emulate the EEH error handling
> > (and may never do).
> Some comments,
> 
> The model should initialize *two* default PHBs per chip for all 
> processor flavors. Extra ones would come with user creatable PHBs,
> which can be added later.
> 
> Can someone take a closer look at pnv_phb3_pci_create() ? This is 
> where the PCI busses, bridges, etc are created.

I had a bit of a look, and made a comment, though I think it's mostly
sane.

> PBCQ is a logic unit acting as a bridge on the PowerBus but the model 
> might be adding some extra unnecessary complexity. may be the PHB3 and 
> the PBCQ models could be merged. This is minor I think.
> 
> In which sub-tree should this patchset be merged ? PCI or PPC ?

Through the ppc tree, I think.  Although review by the PCI folks would
be great if we can get it.

> 
> Thanks,
> 
> C.  
>  
> 
> > Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > [clg: rewrote the QOM models
> >       misc fixes
> >       lots of love and care]
> > Signed-off-by: Cédric Le Goater <clg@kaod.org>
> > ---
> >  default-configs/ppc64-softmmu.mak   |    1 +
> >  include/hw/pci-host/pnv_phb3.h      |  161 +++++
> >  include/hw/pci-host/pnv_phb3_regs.h |  467 ++++++++++++++
> >  include/hw/ppc/pnv.h                |   22 +
> >  include/hw/ppc/pnv_xscom.h          |    9 +
> >  include/hw/ppc/xics.h               |    1 +
> >  hw/intc/xics.c                      |    2 +-
> >  hw/pci-host/pnv_phb3.c              | 1219 +++++++++++++++++++++++++++++++++++
> >  hw/pci-host/pnv_phb3_msi.c          |  316 +++++++++
> >  hw/pci-host/pnv_phb3_pbcq.c         |  347 ++++++++++
> >  hw/ppc/pnv.c                        |   75 ++-
> >  hw/ppc/pnv_xscom.c                  |    6 +-
> >  hw/pci-host/Makefile.objs           |    1 +
> >  13 files changed, 2622 insertions(+), 5 deletions(-)
> >  create mode 100644 include/hw/pci-host/pnv_phb3.h
> >  create mode 100644 include/hw/pci-host/pnv_phb3_regs.h
> >  create mode 100644 hw/pci-host/pnv_phb3.c
> >  create mode 100644 hw/pci-host/pnv_phb3_msi.c
> >  create mode 100644 hw/pci-host/pnv_phb3_pbcq.c
> > 
> > diff --git a/default-configs/ppc64-softmmu.mak b/default-configs/ppc64-softmmu.mak
> > index b94af6c7c62a..deebba2b044a 100644
> > --- a/default-configs/ppc64-softmmu.mak
> > +++ b/default-configs/ppc64-softmmu.mak
> > @@ -9,6 +9,7 @@ CONFIG_IPMI=y
> >  CONFIG_IPMI_LOCAL=y
> >  CONFIG_IPMI_EXTERN=y
> >  CONFIG_ISA_IPMI_BT=y
> > +CONFIG_PCIE_PORT=y
> >  
> >  # For pSeries
> >  CONFIG_PSERIES=y
> > diff --git a/include/hw/pci-host/pnv_phb3.h b/include/hw/pci-host/pnv_phb3.h
> > new file mode 100644
> > index 000000000000..459994b751ca
> > --- /dev/null
> > +++ b/include/hw/pci-host/pnv_phb3.h
> > @@ -0,0 +1,161 @@
> > +/*
> > + * QEMU PowerPC PowerNV PHB3 model
> > + *
> > + * Copyright (c) 2014-2018, IBM Corporation.
> > + *
> > + * This code is licensed under the GPL version 2 or later. See the
> > + * COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef PCI_HOST_PNV_PHB3_H
> > +#define PCI_HOST_PNV_PHB3_H
> > +
> > +#include "hw/pci/pci_host.h"
> > +#include "hw/ppc/xics.h"
> > +
> > +typedef struct PnvPHB3 PnvPHB3;
> > +typedef struct PnvChip PnvChip;
> > +
> > +/*
> > + * PHB3 XICS Source for MSIs
> > + */
> > +#define TYPE_PHB3_MSI "phb3-msi"
> > +#define PHB3_MSI(obj) OBJECT_CHECK(Phb3MsiState, (obj), TYPE_PHB3_MSI)
> > +
> > +#define PHB3_MAX_MSI     2048
> > +
> > +typedef struct Phb3MsiState {
> > +    ICSState ics;
> > +
> > +    PnvPHB3 *phb;
> > +    uint64_t rba[PHB3_MAX_MSI / 64];
> > +    uint32_t rba_sum;
> > +} Phb3MsiState;
> > +
> > +void pnv_phb3_msi_update_config(Phb3MsiState *msis, uint32_t base,
> > +                                uint32_t count);
> > +void pnv_phb3_msi_send(Phb3MsiState *msis, uint64_t addr, uint16_t data,
> > +                       int32_t dev_pe);
> > +void pnv_phb3_msi_ffi(Phb3MsiState *msis, uint64_t val);
> > +
> > +
> > +/* We have one such address space wrapper per possible device
> > + * under the PHB since they need to be assigned statically at
> > + * qemu device creation time. The relationship to a PE is done
> > + * later dynamically. This means we can potentially create a lot
> > + * of these guys. Q35 stores them as some kind of radix tree but
> > + * we never really need to do fast lookups so instead we simply
> > + * keep a QLIST of them for now, we can add the radix if needed
> > + * later on.
> > + *
> > + * We do cache the PE number to speed things up a bit though.
> > + */
> > +typedef struct PnvPhb3DMASpace {
> > +    PCIBus *bus;
> > +    uint8_t devfn;
> > +    int pe_num;         /* Cached PE number */
> > +#define PHB_INVALID_PE (-1)
> > +    PnvPHB3 *phb;
> > +    AddressSpace dma_as;
> > +    IOMMUMemoryRegion dma_mr;
> > +    MemoryRegion msi32_mr;
> > +    MemoryRegion msi64_mr;
> > +    bool msi32_mapped;
> > +    bool msi64_mapped;
> > +    QLIST_ENTRY(PnvPhb3DMASpace) list;
> > +} PnvPhb3DMASpace;
> > +
> > +/*
> > + * PHB3 Power Bus Common Queue
> > + */
> > +#define TYPE_PNV_PBCQ "pnv-pbcq"
> > +#define PNV_PBCQ(obj) OBJECT_CHECK(PnvPBCQState, (obj), TYPE_PNV_PBCQ)
> > +
> > +typedef struct PnvPBCQState {
> > +    DeviceState parent;
> > +
> > +    uint32_t nest_xbase;
> > +    uint32_t spci_xbase;
> > +    uint32_t pci_xbase;
> > +#define PBCQ_NEST_REGS_COUNT    0x46
> > +#define PBCQ_PCI_REGS_COUNT     0x15
> > +#define PBCQ_SPCI_REGS_COUNT    0x5
> > +
> > +    uint64_t nest_regs[PBCQ_NEST_REGS_COUNT];
> > +    uint64_t spci_regs[PBCQ_SPCI_REGS_COUNT];
> > +    uint64_t pci_regs[PBCQ_PCI_REGS_COUNT];
> > +    MemoryRegion mmbar0;
> > +    MemoryRegion mmbar1;
> > +    MemoryRegion phbbar;
> > +    bool mmio0_mapped;
> > +    bool mmio1_mapped;
> > +    bool phb_mapped;
> > +    uint64_t mmio0_base;
> > +    uint64_t mmio0_size;
> > +    uint64_t mmio1_base;
> > +    uint64_t mmio1_size;
> > +    PnvPHB3 *phb;
> > +
> > +    MemoryRegion xscom_nest_regs;
> > +    MemoryRegion xscom_pci_regs;
> > +    MemoryRegion xscom_spci_regs;
> > +} PnvPBCQState;
> > +
> > +/*
> > + * PHB3 PCI Host Bridge for PowerNV machines (POWER8)
> > + */
> > +#define TYPE_PNV_PHB3 "pnv-phb3"
> > +#define PNV_PHB3(obj) OBJECT_CHECK(PnvPHB3, (obj), TYPE_PNV_PHB3)
> > +
> > +#define PNV_PHB3_NUM_M64      16
> > +#define PNV_PHB3_NUM_REGS     (0x1000 >> 3)
> > +#define PNV_PHB3_NUM_LSI      8
> > +#define PNV_PHB3_NUM_PE       256
> > +
> > +#define PCI_MMIO_TOTAL_SIZE   (0x1ull << 60)
> > +
> > +struct PnvPHB3 {
> > +    PCIHostState parent_obj;
> > +
> > +    uint32_t chip_id;
> > +    uint32_t phb_id;
> > +    char bus_path[8];
> > +
> > +    uint64_t regs[PNV_PHB3_NUM_REGS];
> > +    MemoryRegion mr_regs;
> > +
> > +    MemoryRegion mr_m32;
> > +    MemoryRegion mr_m64[PNV_PHB3_NUM_M64];
> > +    bool regs_mapped;
> > +    bool m32_mapped;
> > +    bool m64_mapped[PNV_PHB3_NUM_M64];
> > +    MemoryRegion pci_mmio;
> > +    MemoryRegion pci_io;
> > +
> > +    uint64_t ioda_LIST[8];
> > +    uint64_t ioda_LXIVT[8];
> > +    uint64_t ioda_TVT[512];
> > +    uint64_t ioda_M64BT[16];
> > +    uint64_t ioda_MDT[256];
> > +    uint64_t ioda_PEEV[4];
> > +
> > +    uint32_t total_irq;
> > +    ICSState lsis;
> > +    Phb3MsiState msis;
> > +
> > +    PnvPBCQState pbcq;
> > +
> > +    QLIST_HEAD(, PnvPhb3DMASpace) dma_spaces;
> > +};
> > +
> > +#define TYPE_PNV_PHB3_RC "pnv-phb3-rc"
> > +
> > +#define TYPE_PNV_PHB3_ROOT_BUS "pnv-phb3-root-bus"
> > +
> > +uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size);
> > +void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size);
> > +void pnv_phb3_update_regions(PnvPHB3 *phb);
> > +void pnv_phb3_remap_irqs(PnvPHB3 *phb);
> > +
> > +
> > +#endif /* PCI_HOST_PNV_PHB3_H */
> > diff --git a/include/hw/pci-host/pnv_phb3_regs.h b/include/hw/pci-host/pnv_phb3_regs.h
> > new file mode 100644
> > index 000000000000..4ddbb5559eb5
> > --- /dev/null
> > +++ b/include/hw/pci-host/pnv_phb3_regs.h
> > @@ -0,0 +1,467 @@
> > +/* Copyright (c) 2013-2018, IBM Corporation.
> > + *
> > + * Licensed under the Apache License, Version 2.0 (the "License");
> > + * you may not use this file except in compliance with the License.
> > + * You may obtain a copy of the License at
> > + *
> > + *      http://www.apache.org/licenses/LICENSE-2.0
> > + *
> > + * Unless required by applicable law or agreed to in writing, software
> > + * distributed under the License is distributed on an "AS IS" BASIS,
> > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > + * implied.
> > + *
> > + * See the License for the specific language governing permissions and
> > + * limitations under the License.
> > + */
> > +
> > +#ifndef PCI_HOST_PNV_PHB3_REGS_H
> > +#define PCI_HOST_PNV_PHB3_REGS_H
> > +
> > +/*
> > + * PBCQ XSCOM registers
> > + */
> > +
> > +#define PBCQ_NEST_IRSN_COMPARE  0x1a
> > +#define PBCQ_NEST_IRSN_COMP           PPC_BITMASK(0, 18)
> > +#define PBCQ_NEST_IRSN_MASK     0x1b
> > +#define PBCQ_NEST_LSI_SRC_ID    0x1f
> > +#define   PBCQ_NEST_LSI_SRC           PPC_BITMASK(0, 7)
> > +#define PBCQ_NEST_REGS_COUNT    0x46
> > +#define PBCQ_NEST_MMIO_BAR0     0x40
> > +#define PBCQ_NEST_MMIO_BAR1     0x41
> > +#define PBCQ_NEST_PHB_BAR       0x42
> > +#define PBCQ_NEST_MMIO_MASK0    0x43
> > +#define PBCQ_NEST_MMIO_MASK1    0x44
> > +#define PBCQ_NEST_BAR_EN        0x45
> > +#define   PBCQ_NEST_BAR_EN_MMIO0    PPC_BIT(0)
> > +#define   PBCQ_NEST_BAR_EN_MMIO1    PPC_BIT(1)
> > +#define   PBCQ_NEST_BAR_EN_PHB      PPC_BIT(2)
> > +#define   PBCQ_NEST_BAR_EN_IRSN_RX  PPC_BIT(3)
> > +#define   PBCQ_NEST_BAR_EN_IRSN_TX  PPC_BIT(4)
> > +
> > +#define PBCQ_PCI_REGS_COUNT     0x15
> > +#define PBCQ_PCI_BAR2           0x0b
> > +
> > +#define PBCQ_SPCI_REGS_COUNT    0x5
> > +#define PBCQ_SPCI_ASB_ADDR      0x0
> > +#define PBCQ_SPCI_ASB_STATUS    0x1
> > +#define PBCQ_SPCI_ASB_DATA      0x2
> > +#define PBCQ_SPCI_AIB_CAPP_EN   0x3
> > +#define PBCQ_SPCI_CAPP_SEC_TMR  0x4
> > +
> > +/*
> > + * PHB MMIO registers
> > + */
> > +
> > +/* PHB Fundamental register set A */
> > +#define PHB_LSI_SOURCE_ID               0x100
> > +#define   PHB_LSI_SRC_ID                PPC_BITMASK(5, 12)
> > +#define PHB_DMA_CHAN_STATUS             0x110
> > +#define   PHB_DMA_CHAN_ANY_ERR          PPC_BIT(27)
> > +#define   PHB_DMA_CHAN_ANY_ERR1         PPC_BIT(28)
> > +#define   PHB_DMA_CHAN_ANY_FREEZE       PPC_BIT(29)
> > +#define PHB_CPU_LOADSTORE_STATUS        0x120
> > +#define   PHB_CPU_LS_ANY_ERR            PPC_BIT(27)
> > +#define   PHB_CPU_LS_ANY_ERR1           PPC_BIT(28)
> > +#define   PHB_CPU_LS_ANY_FREEZE         PPC_BIT(29)
> > +#define PHB_DMA_MSI_NODE_ID             0x128
> > +#define   PHB_DMAMSI_NID_FIXED          PPC_BIT(0)
> > +#define   PHB_DMAMSI_NID_MASK           PPC_BITMASK(24, 31)
> > +#define   PHB_DMAMSI_NID_LSH            PPC_BITLSHIFT(31)
> > +#define PHB_CONFIG_DATA                 0x130
> > +#define PHB_LOCK0                       0x138
> > +#define PHB_CONFIG_ADDRESS              0x140
> > +#define   PHB_CA_ENABLE                 PPC_BIT(0)
> > +#define   PHB_CA_BUS_MASK               PPC_BITMASK(4, 11)
> > +#define   PHB_CA_BUS_LSH                PPC_BITLSHIFT(11)
> > +#define   PHB_CA_DEV_MASK               PPC_BITMASK(12, 16)
> > +#define   PHB_CA_DEV_LSH                PPC_BITLSHIFT(16)
> > +#define   PHB_CA_FUNC_MASK              PPC_BITMASK(17, 19)
> > +#define   PHB_CA_FUNC_LSH               PPC_BITLSHIFT(19)
> > +#define   PHB_CA_REG_MASK               PPC_BITMASK(20, 31)
> > +#define   PHB_CA_REG_LSH                PPC_BITLSHIFT(31)
> > +#define   PHB_CA_PE_MASK                PPC_BITMASK(40, 47)
> > +#define   PHB_CA_PE_LSH                 PPC_BITLSHIFT(47)
> > +#define PHB_LOCK1                       0x148
> > +#define PHB_IVT_BAR                     0x150
> > +#define   PHB_IVT_BAR_ENABLE            PPC_BIT(0)
> > +#define   PHB_IVT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 48)
> > +#define   PHB_IVT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(48)
> > +#define   PHB_IVT_LENGTH_MASK           PPC_BITMASK(52, 63)
> > +#define   PHB_IVT_LENGTH_ADDRESS_LSH    PPC_BITLSHIFT(63)
> > +#define PHB_RBA_BAR                     0x158
> > +#define   PHB_RBA_BAR_ENABLE            PPC_BIT(0)
> > +#define   PHB_RBA_BASE_ADDRESS_MASK     PPC_BITMASK(14, 55)
> > +#define   PHB_RBA_BASE_ADDRESS_LSH      PPC_BITLSHIFT(55)
> > +#define PHB_PHB3_CONFIG                 0x160
> > +#define   PHB_PHB3C_64B_TCE_EN          PPC_BIT(2)
> > +#define   PHB_PHB3C_32BIT_MSI_EN        PPC_BIT(8)
> > +#define   PHB_PHB3C_64BIT_MSI_EN        PPC_BIT(14)
> > +#define   PHB_PHB3C_M32_EN              PPC_BIT(16)
> > +#define PHB_RTT_BAR                     0x168
> > +#define   PHB_RTT_BAR_ENABLE            PPC_BIT(0)
> > +#define   PHB_RTT_BASE_ADDRESS_MASK     PPC_BITMASK(14, 46)
> > +#define   PHB_RTT_BASE_ADDRESS_LSH      PPC_BITLSHIFT(46)
> > +#define PHB_PELTV_BAR                   0x188
> > +#define   PHB_PELTV_BAR_ENABLE          PPC_BIT(0)
> > +#define   PHB_PELTV_BASE_ADDRESS_MASK   PPC_BITMASK(14, 50)
> > +#define   PHB_PELTV_BASE_ADDRESS_LSH    PPC_BITLSHIFT(50)
> > +#define PHB_M32_BASE_ADDR               0x190
> > +#define PHB_M32_BASE_MASK               0x198
> > +#define PHB_M32_START_ADDR              0x1a0
> > +#define PHB_PEST_BAR                    0x1a8
> > +#define   PHB_PEST_BAR_ENABLE           PPC_BIT(0)
> > +#define   PHB_PEST_BASE_ADDRESS_MASK    PPC_BITMASK(14, 51)
> > +#define   PHB_PEST_BASE_ADDRESS_LSH     PPC_BITLSHIFT(51)
> > +#define PHB_M64_UPPER_BITS              0x1f0
> > +#define PHB_INTREP_TIMER                0x1f8
> > +#define PHB_DMARD_SYNC                  0x200
> > +#define PHB_RTC_INVALIDATE              0x208
> > +#define   PHB_RTC_INVALIDATE_ALL        PPC_BIT(0)
> > +#define   PHB_RTC_INVALIDATE_RID_MASK   PPC_BITMASK(16, 31)
> > +#define   PHB_RTC_INVALIDATE_RID_LSH    PPC_BITLSHIFT(31)
> > +#define PHB_TCE_KILL                    0x210
> > +#define   PHB_TCE_KILL_ALL              PPC_BIT(0)
> > +#define PHB_TCE_SPEC_CTL                0x218
> > +#define PHB_IODA_ADDR                   0x220
> > +#define   PHB_IODA_AD_AUTOINC           PPC_BIT(0)
> > +#define   PHB_IODA_AD_TSEL              PPC_BITMASK(11, 15)
> > +#define   PHB_IODA_AD_TADR              PPC_BITMASK(55, 63)
> > +#define PHB_IODA_DATA0                  0x228
> > +#define PHB_FFI_REQUEST                 0x238
> > +#define   PHB_FFI_LOCK_CLEAR            PPC_BIT(3)
> > +#define   PHB_FFI_REQUEST_ISN_MASK      PPC_BITMASK(49, 59)
> > +#define   PHB_FFI_REQUEST_ISN_LSH       PPC_BITLSHIFT(59)
> > +#define PHB_FFI_LOCK                    0x240
> > +#define   PHB_FFI_LOCK_STATE            PPC_BIT(0)
> > +#define PHB_XIVE_UPDATE                 0x248 /* Broken in DD1 */
> > +#define PHB_PHB3_GEN_CAP                0x250
> > +#define PHB_PHB3_TCE_CAP                0x258
> > +#define PHB_PHB3_IRQ_CAP                0x260
> > +#define PHB_PHB3_EEH_CAP                0x268
> > +#define PHB_IVC_INVALIDATE              0x2a0
> > +#define   PHB_IVC_INVALIDATE_ALL        PPC_BIT(0)
> > +#define   PHB_IVC_INVALIDATE_SID_MASK   PPC_BITMASK(16, 31)
> > +#define   PHB_IVC_INVALIDATE_SID_LSH    PPC_BITLSHIFT(31)
> > +#define PHB_IVC_UPDATE                  0x2a8
> > +#define   PHB_IVC_UPDATE_ENABLE_P       PPC_BIT(0)
> > +#define   PHB_IVC_UPDATE_ENABLE_Q       PPC_BIT(1)
> > +#define   PHB_IVC_UPDATE_ENABLE_SERVER  PPC_BIT(2)
> > +#define   PHB_IVC_UPDATE_ENABLE_PRI     PPC_BIT(3)
> > +#define   PHB_IVC_UPDATE_ENABLE_GEN     PPC_BIT(4)
> > +#define   PHB_IVC_UPDATE_ENABLE_CON     PPC_BIT(5)
> > +#define   PHB_IVC_UPDATE_GEN_MATCH_MASK PPC_BITMASK(6, 7)
> > +#define   PHB_IVC_UPDATE_GEN_MATCH_LSH  PPC_BITLSHIFT(7)
> > +#define   PHB_IVC_UPDATE_SERVER_MASK    PPC_BITMASK(8, 23)
> > +#define   PHB_IVC_UPDATE_SERVER_LSH     PPC_BITLSHIFT(23)
> > +#define   PHB_IVC_UPDATE_PRI_MASK       PPC_BITMASK(24, 31)
> > +#define   PHB_IVC_UPDATE_PRI_LSH        PPC_BITLSHIFT(31)
> > +#define   PHB_IVC_UPDATE_GEN_MASK       PPC_BITMASK(32, 33)
> > +#define   PHB_IVC_UPDATE_GEN_LSH        PPC_BITLSHIFT(33)
> > +#define   PHB_IVC_UPDATE_P_MASK         PPC_BITMASK(34, 34)
> > +#define   PHB_IVC_UPDATE_P_LSH          PPC_BITLSHIFT(34)
> > +#define   PHB_IVC_UPDATE_Q_MASK         PPC_BITMASK(35, 35)
> > +#define   PHB_IVC_UPDATE_Q_LSH          PPC_BITLSHIFT(35)
> > +#define   PHB_IVC_UPDATE_SID_MASK       PPC_BITMASK(48, 63)
> > +#define   PHB_IVC_UPDATE_SID_LSH        PPC_BITLSHIFT(63)
> > +#define PHB_PAPR_ERR_INJ_CTL            0x2b0
> > +#define   PHB_PAPR_ERR_INJ_CTL_INB      PPC_BIT(0)
> > +#define   PHB_PAPR_ERR_INJ_CTL_OUTB     PPC_BIT(1)
> > +#define   PHB_PAPR_ERR_INJ_CTL_STICKY   PPC_BIT(2)
> > +#define   PHB_PAPR_ERR_INJ_CTL_CFG      PPC_BIT(3)
> > +#define   PHB_PAPR_ERR_INJ_CTL_RD       PPC_BIT(4)
> > +#define   PHB_PAPR_ERR_INJ_CTL_WR       PPC_BIT(5)
> > +#define   PHB_PAPR_ERR_INJ_CTL_FREEZE   PPC_BIT(6)
> > +#define PHB_PAPR_ERR_INJ_ADDR           0x2b8
> > +#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_MASK       PPC_BITMASK(16, 63)
> > +#define   PHB_PAPR_ERR_INJ_ADDR_MMIO_LSH        PPC_BITLSHIFT(63)
> > +#define PHB_PAPR_ERR_INJ_MASK           0x2c0
> > +#define   PHB_PAPR_ERR_INJ_MASK_CFG_MASK        PPC_BITMASK(4, 11)
> > +#define   PHB_PAPR_ERR_INJ_MASK_CFG_LSH         PPC_BITLSHIFT(11)
> > +#define   PHB_PAPR_ERR_INJ_MASK_MMIO_MASK       PPC_BITMASK(16, 63)
> > +#define   PHB_PAPR_ERR_INJ_MASK_MMIO_LSH        PPC_BITLSHIFT(63)
> > +#define PHB_ETU_ERR_SUMMARY             0x2c8
> > +
> > +/*  UTL registers */
> > +#define UTL_SYS_BUS_CONTROL             0x400
> > +#define UTL_STATUS                      0x408
> > +#define UTL_SYS_BUS_AGENT_STATUS        0x410
> > +#define UTL_SYS_BUS_AGENT_ERR_SEVERITY  0x418
> > +#define UTL_SYS_BUS_AGENT_IRQ_EN        0x420
> > +#define UTL_SYS_BUS_BURST_SZ_CONF       0x440
> > +#define UTL_REVISION_ID                 0x448
> > +#define UTL_BCLK_DOMAIN_DBG1            0x460
> > +#define UTL_BCLK_DOMAIN_DBG2            0x468
> > +#define UTL_BCLK_DOMAIN_DBG3            0x470
> > +#define UTL_BCLK_DOMAIN_DBG4            0x478
> > +#define UTL_BCLK_DOMAIN_DBG5            0x480
> > +#define UTL_BCLK_DOMAIN_DBG6            0x488
> > +#define UTL_OUT_POST_HDR_BUF_ALLOC      0x4c0
> > +#define UTL_OUT_POST_DAT_BUF_ALLOC      0x4d0
> > +#define UTL_IN_POST_HDR_BUF_ALLOC       0x4e0
> > +#define UTL_IN_POST_DAT_BUF_ALLOC       0x4f0
> > +#define UTL_OUT_NP_BUF_ALLOC            0x500
> > +#define UTL_IN_NP_BUF_ALLOC             0x510
> > +#define UTL_PCIE_TAGS_ALLOC             0x520
> > +#define UTL_GBIF_READ_TAGS_ALLOC        0x530
> > +#define UTL_PCIE_PORT_CONTROL           0x540
> > +#define UTL_PCIE_PORT_STATUS            0x548
> > +#define UTL_PCIE_PORT_ERROR_SEV         0x550
> > +#define UTL_PCIE_PORT_IRQ_EN            0x558
> > +#define UTL_RC_STATUS                   0x560
> > +#define UTL_RC_ERR_SEVERITY             0x568
> > +#define UTL_RC_IRQ_EN                   0x570
> > +#define UTL_EP_STATUS                   0x578
> > +#define UTL_EP_ERR_SEVERITY             0x580
> > +#define UTL_EP_ERR_IRQ_EN               0x588
> > +#define UTL_PCI_PM_CTRL1                0x590
> > +#define UTL_PCI_PM_CTRL2                0x598
> > +#define UTL_GP_CTL1                     0x5a0
> > +#define UTL_GP_CTL2                     0x5a8
> > +#define UTL_PCLK_DOMAIN_DBG1            0x5b0
> > +#define UTL_PCLK_DOMAIN_DBG2            0x5b8
> > +#define UTL_PCLK_DOMAIN_DBG3            0x5c0
> > +#define UTL_PCLK_DOMAIN_DBG4            0x5c8
> > +
> > +/* PCI-E Stack registers */
> > +#define PHB_PCIE_SYSTEM_CONFIG          0x600
> > +#define PHB_PCIE_BUS_NUMBER             0x608
> > +#define PHB_PCIE_SYSTEM_TEST            0x618
> > +#define PHB_PCIE_LINK_MANAGEMENT        0x630
> > +#define   PHB_PCIE_LM_LINK_ACTIVE       PPC_BIT(8)
> > +#define PHB_PCIE_DLP_TRAIN_CTL          0x640
> > +#define   PHB_PCIE_DLP_TCTX_DISABLE     PPC_BIT(1)
> > +#define   PHB_PCIE_DLP_TCRX_DISABLED    PPC_BIT(16)
> > +#define   PHB_PCIE_DLP_INBAND_PRESENCE  PPC_BIT(19)
> > +#define   PHB_PCIE_DLP_TC_DL_LINKUP     PPC_BIT(21)
> > +#define   PHB_PCIE_DLP_TC_DL_PGRESET    PPC_BIT(22)
> > +#define   PHB_PCIE_DLP_TC_DL_LINKACT    PPC_BIT(23)
> > +#define PHB_PCIE_SLOP_LOOPBACK_STATUS   0x648
> > +#define PHB_PCIE_SYS_LINK_INIT          0x668
> > +#define PHB_PCIE_UTL_CONFIG             0x670
> > +#define PHB_PCIE_DLP_CONTROL            0x678
> > +#define PHB_PCIE_UTL_ERRLOG1            0x680
> > +#define PHB_PCIE_UTL_ERRLOG2            0x688
> > +#define PHB_PCIE_UTL_ERRLOG3            0x690
> > +#define PHB_PCIE_UTL_ERRLOG4            0x698
> > +#define PHB_PCIE_DLP_ERRLOG1            0x6a0
> > +#define PHB_PCIE_DLP_ERRLOG2            0x6a8
> > +#define PHB_PCIE_DLP_ERR_STATUS         0x6b0
> > +#define PHB_PCIE_DLP_ERR_COUNTERS       0x6b8
> > +#define PHB_PCIE_UTL_ERR_INJECT         0x6c0
> > +#define PHB_PCIE_TLDLP_ERR_INJECT       0x6c8
> > +#define PHB_PCIE_LANE_EQ_CNTL0          0x6d0
> > +#define PHB_PCIE_LANE_EQ_CNTL1          0x6d8
> > +#define PHB_PCIE_LANE_EQ_CNTL2          0x6e0
> > +#define PHB_PCIE_LANE_EQ_CNTL3          0x6e8
> > +#define PHB_PCIE_STRAPPING              0x700
> > +
> > +/* Fundamental register set B */
> > +#define PHB_VERSION                     0x800
> > +#define PHB_RESET                       0x808
> > +#define PHB_CONTROL                     0x810
> > +#define   PHB_CTRL_IVE_128_BYTES        PPC_BIT(24)
> > +#define PHB_AIB_RX_CRED_INIT_TIMER      0x818
> > +#define PHB_AIB_RX_CMD_CRED             0x820
> > +#define PHB_AIB_RX_DATA_CRED            0x828
> > +#define PHB_AIB_TX_CMD_CRED             0x830
> > +#define PHB_AIB_TX_DATA_CRED            0x838
> > +#define PHB_AIB_TX_CHAN_MAPPING         0x840
> > +#define PHB_AIB_TAG_ENABLE              0x858
> > +#define PHB_AIB_FENCE_CTRL              0x860
> > +#define PHB_TCE_TAG_ENABLE              0x868
> > +#define PHB_TCE_WATERMARK               0x870
> > +#define PHB_TIMEOUT_CTRL1               0x878
> > +#define PHB_TIMEOUT_CTRL2               0x880
> > +#define PHB_QUIESCE_DMA_G               0x888
> > +#define PHB_AIB_TAG_STATUS              0x900
> > +#define PHB_TCE_TAG_STATUS              0x908
> > +
> > +/* FIR & Error registers */
> > +#define PHB_LEM_FIR_ACCUM               0xc00
> > +#define PHB_LEM_FIR_AND_MASK            0xc08
> > +#define PHB_LEM_FIR_OR_MASK             0xc10
> > +#define PHB_LEM_ERROR_MASK              0xc18
> > +#define PHB_LEM_ERROR_AND_MASK          0xc20
> > +#define PHB_LEM_ERROR_OR_MASK           0xc28
> > +#define PHB_LEM_ACTION0                 0xc30
> > +#define PHB_LEM_ACTION1                 0xc38
> > +#define PHB_LEM_WOF                     0xc40
> > +#define PHB_ERR_STATUS                  0xc80
> > +#define PHB_ERR1_STATUS                 0xc88
> > +#define PHB_ERR_INJECT                  0xc90
> > +#define PHB_ERR_LEM_ENABLE              0xc98
> > +#define PHB_ERR_IRQ_ENABLE              0xca0
> > +#define PHB_ERR_FREEZE_ENABLE           0xca8
> > +#define PHB_ERR_AIB_FENCE_ENABLE        0xcb0
> > +#define PHB_ERR_LOG_0                   0xcc0
> > +#define PHB_ERR_LOG_1                   0xcc8
> > +#define PHB_ERR_STATUS_MASK             0xcd0
> > +#define PHB_ERR1_STATUS_MASK            0xcd8
> > +
> > +#define PHB_OUT_ERR_STATUS              0xd00
> > +#define PHB_OUT_ERR1_STATUS             0xd08
> > +#define PHB_OUT_ERR_INJECT              0xd10
> > +#define PHB_OUT_ERR_LEM_ENABLE          0xd18
> > +#define PHB_OUT_ERR_IRQ_ENABLE          0xd20
> > +#define PHB_OUT_ERR_FREEZE_ENABLE       0xd28
> > +#define PHB_OUT_ERR_AIB_FENCE_ENABLE    0xd30
> > +#define PHB_OUT_ERR_LOG_0               0xd40
> > +#define PHB_OUT_ERR_LOG_1               0xd48
> > +#define PHB_OUT_ERR_STATUS_MASK         0xd50
> > +#define PHB_OUT_ERR1_STATUS_MASK        0xd58
> > +
> > +#define PHB_INA_ERR_STATUS              0xd80
> > +#define PHB_INA_ERR1_STATUS             0xd88
> > +#define PHB_INA_ERR_INJECT              0xd90
> > +#define PHB_INA_ERR_LEM_ENABLE          0xd98
> > +#define PHB_INA_ERR_IRQ_ENABLE          0xda0
> > +#define PHB_INA_ERR_FREEZE_ENABLE       0xda8
> > +#define PHB_INA_ERR_AIB_FENCE_ENABLE    0xdb0
> > +#define PHB_INA_ERR_LOG_0               0xdc0
> > +#define PHB_INA_ERR_LOG_1               0xdc8
> > +#define PHB_INA_ERR_STATUS_MASK         0xdd0
> > +#define PHB_INA_ERR1_STATUS_MASK        0xdd8
> > +
> > +#define PHB_INB_ERR_STATUS              0xe00
> > +#define PHB_INB_ERR1_STATUS             0xe08
> > +#define PHB_INB_ERR_INJECT              0xe10
> > +#define PHB_INB_ERR_LEM_ENABLE          0xe18
> > +#define PHB_INB_ERR_IRQ_ENABLE          0xe20
> > +#define PHB_INB_ERR_FREEZE_ENABLE       0xe28
> > +#define PHB_INB_ERR_AIB_FENCE_ENABLE    0xe30
> > +#define PHB_INB_ERR_LOG_0               0xe40
> > +#define PHB_INB_ERR_LOG_1               0xe48
> > +#define PHB_INB_ERR_STATUS_MASK         0xe50
> > +#define PHB_INB_ERR1_STATUS_MASK        0xe58
> > +
> > +/* Performance monitor & Debug registers */
> > +#define PHB_TRACE_CONTROL               0xf80
> > +#define PHB_PERFMON_CONFIG              0xf88
> > +#define PHB_PERFMON_CTR0                0xf90
> > +#define PHB_PERFMON_CTR1                0xf98
> > +#define PHB_PERFMON_CTR2                0xfa0
> > +#define PHB_PERFMON_CTR3                0xfa8
> > +#define PHB_HOTPLUG_OVERRIDE            0xfb0
> > +#define   PHB_HPOVR_FORCE_RESAMPLE      PPC_BIT(9)
> > +#define   PHB_HPOVR_PRESENCE_A          PPC_BIT(10)
> > +#define   PHB_HPOVR_PRESENCE_B          PPC_BIT(11)
> > +#define   PHB_HPOVR_LINK_ACTIVE         PPC_BIT(12)
> > +#define   PHB_HPOVR_LINK_BIFURCATED     PPC_BIT(13)
> > +#define   PHB_HPOVR_LINK_LANE_SWAPPED   PPC_BIT(14)
> > +
> > +/*
> > + * IODA2 on-chip tables
> > + */
> > +
> > +#define IODA2_TBL_LIST          1
> > +#define IODA2_TBL_LXIVT         2
> > +#define IODA2_TBL_IVC_CAM       3
> > +#define IODA2_TBL_RBA           4
> > +#define IODA2_TBL_RCAM          5
> > +#define IODA2_TBL_MRT           6
> > +#define IODA2_TBL_PESTA         7
> > +#define IODA2_TBL_PESTB         8
> > +#define IODA2_TBL_TVT           9
> > +#define IODA2_TBL_TCAM          10
> > +#define IODA2_TBL_TDR           11
> > +#define IODA2_TBL_M64BT         16
> > +#define IODA2_TBL_M32DT         17
> > +#define IODA2_TBL_PEEV          20
> > +
> > +/* LXIVT */
> > +#define IODA2_LXIVT_SERVER              PPC_BITMASK(8, 23)
> > +#define IODA2_LXIVT_PRIORITY            PPC_BITMASK(24, 31)
> > +#define IODA2_LXIVT_NODE_ID             PPC_BITMASK(56, 63)
> > +
> > +/* IVT */
> > +#define IODA2_IVT_SERVER                PPC_BITMASK(0, 23)
> > +#define IODA2_IVT_PRIORITY              PPC_BITMASK(24, 31)
> > +#define IODA2_IVT_GEN                   PPC_BITMASK(37, 38)
> > +#define IODA2_IVT_P                     PPC_BITMASK(39, 39)
> > +#define IODA2_IVT_Q                     PPC_BITMASK(47, 47)
> > +#define IODA2_IVT_PE                    PPC_BITMASK(48, 63)
> > +
> > +/* TVT */
> > +#define IODA2_TVT_TABLE_ADDR            PPC_BITMASK(0, 47)
> > +#define IODA2_TVT_NUM_LEVELS            PPC_BITMASK(48, 50)
> > +#define   IODA2_TVE_1_LEVEL     0
> > +#define   IODA2_TVE_2_LEVELS    1
> > +#define   IODA2_TVE_3_LEVELS    2
> > +#define   IODA2_TVE_4_LEVELS    3
> > +#define   IODA2_TVE_5_LEVELS    4
> > +#define IODA2_TVT_TCE_TABLE_SIZE        PPC_BITMASK(51, 55)
> > +#define IODA2_TVT_IO_PSIZE              PPC_BITMASK(59, 63)
> > +
> > +/* PESTA */
> > +#define IODA2_PESTA_MMIO_FROZEN         PPC_BIT(0)
> > +
> > +/* PESTB */
> > +#define IODA2_PESTB_DMA_STOPPED         PPC_BIT(0)
> > +
> > +/* M32DT */
> > +#define IODA2_M32DT_PE_MASK             PPC_BITMASK(8, 15)
> > +#define IODA2_M32DT_PE_LSH              PPC_BITLSHIFT(15)
> > +
> > +/* M64BT */
> > +#define IODA2_M64BT_ENABLE              PPC_BIT(0)
> > +#define IODA2_M64BT_SINGLE_PE           PPC_BIT(1)
> > +#define IODA2_M64BT_BASE                PPC_BITMASK(2, 31)
> > +#define IODA2_M64BT_MASK                PPC_BITMASK(34, 63)
> > +#define IODA2_M64BT_SINGLE_BASE_MASK    PPC_BITMASK(2, 26)
> > +#define IODA2_M64BT_SINGLE_BASE_LSH     PPC_BITLSHIFT(26)
> > +#define IODA2_M64BT_PE_HI_MASK          PPC_BITMASK(27, 31)
> > +#define IODA2_M64BT_PE_HI_LSH           PPC_BITLSHIFT(31)
> > +#define IODA2_M64BT_SINGLE_MASK_MASK    PPC_BITMASK(34, 58)
> > +#define IODA2_M64BT_SINGLE_MASK_LSH     PPC_BITLSHIFT(58)
> > +#define IODA2_M64BT_PE_LOW_MASK         PPC_BITMASK(59, 63)
> > +#define IODA2_M64BT_PE_LOW_LSH          PPC_BITLSHIFT(63)
> > +
> > +/*
> > + * IODA2 in-memory tables
> > + */
> > +
> > +/* PEST
> > + *
> > + * 2x8 bytes entries, PEST0 and PEST1
> > + */
> > +
> > +#define IODA2_PEST0_MMIO_CAUSE          PPC_BIT(2)
> > +#define IODA2_PEST0_CFG_READ            PPC_BIT(3)
> > +#define IODA2_PEST0_CFG_WRITE           PPC_BIT(4)
> > +#define IODA2_PEST0_TTYPE_MASK          PPC_BITMASK(5, 7)
> > +#define IODA2_PEST0_TTYPE_LSH           PPC_BITLSHIFT(7)
> > +#define   PEST_TTYPE_DMA_WRITE          0
> > +#define   PEST_TTYPE_MSI                1
> > +#define   PEST_TTYPE_DMA_READ           2
> > +#define   PEST_TTYPE_DMA_READ_RESP      3
> > +#define   PEST_TTYPE_MMIO_LOAD          4
> > +#define   PEST_TTYPE_MMIO_STORE         5
> > +#define   PEST_TTYPE_OTHER              7
> > +#define IODA2_PEST0_CA_RETURN           PPC_BIT(8)
> > +#define IODA2_PEST0_UTL_RTOS_TIMEOUT    PPC_BIT(8) /* Same bit as CA return */
> > +#define IODA2_PEST0_UR_RETURN           PPC_BIT(9)
> > +#define IODA2_PEST0_UTL_NONFATAL        PPC_BIT(10)
> > +#define IODA2_PEST0_UTL_FATAL           PPC_BIT(11)
> > +#define IODA2_PEST0_PARITY_UE           PPC_BIT(13)
> > +#define IODA2_PEST0_UTL_CORRECTABLE     PPC_BIT(14)
> > +#define IODA2_PEST0_UTL_INTERRUPT       PPC_BIT(15)
> > +#define IODA2_PEST0_MMIO_XLATE          PPC_BIT(16)
> > +#define IODA2_PEST0_IODA2_ERROR         PPC_BIT(16) /* Same bit as MMIO xlate */
> > +#define IODA2_PEST0_TCE_PAGE_FAULT      PPC_BIT(18)
> > +#define IODA2_PEST0_TCE_ACCESS_FAULT    PPC_BIT(19)
> > +#define IODA2_PEST0_DMA_RESP_TIMEOUT    PPC_BIT(20)
> > +#define IODA2_PEST0_AIB_SIZE_INVALID    PPC_BIT(21)
> > +#define IODA2_PEST0_LEM_BIT_MASK        PPC_BITMASK(26, 31)
> > +#define IODA2_PEST0_LEM_BIT_LSH         PPC_BITLSHIFT(31)
> > +#define IODA2_PEST0_RID_MASK            PPC_BITMASK(32, 47)
> > +#define IODA2_PEST0_RID_LSH             PPC_BITLSHIFT(47)
> > +#define IODA2_PEST0_MSI_DATA_MASK       PPC_BITMASK(48, 63)
> > +#define IODA2_PEST0_MSI_DATA_LSH        PPC_BITLSHIFT(63)
> > +
> > +#define IODA2_PEST1_FAIL_ADDR_MASK      PPC_BITMASK(3, 63)
> > +#define IODA2_PEST1_FAIL_ADDR_LSH       PPC_BITLSHIFT(63)
> > +
> > +
> > +#endif /* PCI_HOST_PNV_PHB3_REGS_H */
> > diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
> > index 86d5f54e5459..61af65fc0b76 100644
> > --- a/include/hw/ppc/pnv.h
> > +++ b/include/hw/ppc/pnv.h
> > @@ -25,6 +25,7 @@
> >  #include "hw/ppc/pnv_lpc.h"
> >  #include "hw/ppc/pnv_psi.h"
> >  #include "hw/ppc/pnv_occ.h"
> > +#include "hw/pci-host/pnv_phb3.h"
> >  
> >  #define TYPE_PNV_CHIP "pnv-chip"
> >  #define PNV_CHIP(obj) OBJECT_CHECK(PnvChip, (obj), TYPE_PNV_CHIP)
> > @@ -72,6 +73,9 @@ typedef struct Pnv8Chip {
> >      PnvLpcController lpc;
> >      PnvPsi       psi;
> >      PnvOCC       occ;
> > +
> > +#define PNV8_CHIP_PHB3_MAX 4
> > +    PnvPHB3      phbs[PNV8_CHIP_PHB3_MAX];
> >  } Pnv8Chip;
> >  
> >  #define TYPE_PNV9_CHIP "pnv9-chip"
> > @@ -92,6 +96,7 @@ typedef struct PnvChipClass {
> >      PnvChipType  chip_type;
> >      uint64_t     chip_cfam_id;
> >      uint64_t     cores_mask;
> > +    uint32_t     num_phbs;
> >  
> >      hwaddr       xscom_base;
> >  
> > @@ -171,6 +176,23 @@ static inline bool pnv_is_power9(PnvMachineState *pnv)
> >      return pnv_chip_is_power9(pnv->chips[0]);
> >  }
> >  
> > +/*
> > + * This is used by devices created on the command line to find a chip
> > + * on which to attach to.
> > + */
> > +static inline PnvChip *pnv_get_chip(PnvMachineState *pnv, uint chip_id)
> > +{
> > +    int i;
> > +
> > +    for (i = 0; i < pnv->num_chips; i++) {
> > +        PnvChip *chip = pnv->chips[i];
> > +        if (chip->chip_id == chip_id) {
> > +            return chip;
> > +        }
> > +    }
> > +    return NULL;
> > +}
> > +
> >  #define PNV_FDT_ADDR          0x01000000
> >  #define PNV_TIMEBASE_FREQ     512000000ULL
> >  
> > diff --git a/include/hw/ppc/pnv_xscom.h b/include/hw/ppc/pnv_xscom.h
> > index 255b26a5aaf6..e7ebffa5cf8a 100644
> > --- a/include/hw/ppc/pnv_xscom.h
> > +++ b/include/hw/ppc/pnv_xscom.h
> > @@ -73,6 +73,15 @@ typedef struct PnvXScomInterfaceClass {
> >  #define PNV_XSCOM_OCC_BASE        0x0066000
> >  #define PNV_XSCOM_OCC_SIZE        0x6000
> >  
> > +#define PNV_XSCOM_PBCQ_NEST_BASE  0x2012000
> > +#define PNV_XSCOM_PBCQ_NEST_SIZE  0x46
> > +
> > +#define PNV_XSCOM_PBCQ_PCI_BASE   0x9012000
> > +#define PNV_XSCOM_PBCQ_PCI_SIZE   0x15
> > +
> > +#define PNV_XSCOM_PBCQ_SPCI_BASE  0x9013c00
> > +#define PNV_XSCOM_PBCQ_SPCI_SIZE  0x5
> > +
> >  extern void pnv_xscom_realize(PnvChip *chip, Error **errp);
> >  extern int pnv_dt_xscom(PnvChip *chip, void *fdt, int offset);
> >  
> > diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> > index 6ac8a9392da6..966a996c2eac 100644
> > --- a/include/hw/ppc/xics.h
> > +++ b/include/hw/ppc/xics.h
> > @@ -194,6 +194,7 @@ void icp_set_mfrr(ICPState *icp, uint8_t mfrr);
> >  uint32_t icp_accept(ICPState *ss);
> >  uint32_t icp_ipoll(ICPState *ss, uint32_t *mfrr);
> >  void icp_eoi(ICPState *icp, uint32_t xirr);
> > +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority);
> >  
> >  void ics_simple_write_xive(ICSState *ics, int nr, int server,
> >                             uint8_t priority, uint8_t saved_priority);
> > diff --git a/hw/intc/xics.c b/hw/intc/xics.c
> > index b9f1a3c97214..59e2a5217dcc 100644
> > --- a/hw/intc/xics.c
> > +++ b/hw/intc/xics.c
> > @@ -230,7 +230,7 @@ void icp_eoi(ICPState *icp, uint32_t xirr)
> >      }
> >  }
> >  
> > -static void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
> > +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
> >  {
> >      ICPState *icp = xics_icp_get(ics->xics, server);
> >  
> > diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
> > new file mode 100644
> > index 000000000000..e11993e0e1fb
> > --- /dev/null
> > +++ b/hw/pci-host/pnv_phb3.c
> > @@ -0,0 +1,1219 @@
> > +/*
> > + * QEMU PowerPC PowerNV PHB3 model
> > + *
> > + * Copyright (c) 2014-2018, IBM Corporation.
> > + *
> > + * This code is licensed under the GPL version 2 or later. See the
> > + * COPYING file in the top-level directory.
> > + */
> > +#include "qemu/osdep.h"
> > +#include "qemu/log.h"
> > +#include "qapi/visitor.h"
> > +#include "qapi/error.h"
> > +#include "qemu-common.h"
> > +#include "target/ppc/cpu.h"
> > +#include "hw/pci-host/pnv_phb3_regs.h"
> > +#include "hw/pci-host/pnv_phb3.h"
> > +#include "hw/pci/pci_bridge.h"
> > +#include "hw/pci/pci_bus.h"
> > +#include "hw/pci/pcie_port.h"
> > +#include "hw/ppc/pnv.h"
> > +
> > +static PCIDevice *pnv_phb3_find_cfg_dev(PnvPHB3 *phb)
> > +{
> > +    PCIHostState *pci = PCI_HOST_BRIDGE(phb);
> > +    uint64_t addr = phb->regs[PHB_CONFIG_ADDRESS >> 3];
> > +    uint8_t bus, devfn;
> > +
> > +    if (!(addr >> 63)) {
> > +        return NULL;
> > +    }
> > +    bus = (addr >> 52) & 0xff;
> > +    devfn = (addr >> 44) & 0xff;
> > +
> > +    return pci_find_device(pci->bus, bus, devfn);
> > +}
> > +
> > +/*
> > + * The CONFIG_DATA register expects little endian accesses, but as the
> > + * region is big endian, we have to swap the value.
> > + */
> > +static void pnv_phb3_config_write(PnvPHB3 *phb, unsigned off,
> > +                                  unsigned size, uint64_t val)
> > +{
> > +    uint32_t cfg_addr, limit;
> > +    PCIDevice *pdev;
> > +
> > +    pdev = pnv_phb3_find_cfg_dev(phb);
> > +    if (!pdev) {
> > +        return;
> > +    }
> > +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xfff;
> > +    cfg_addr |= off;
> > +    limit = pci_config_size(pdev);
> > +    if (limit <= cfg_addr) {
> > +        /* conventional pci device can be behind pcie-to-pci bridge.
> > +           256 <= addr < 4K has no effects. */
> > +        return;
> > +    }
> > +    switch (size) {
> > +    case 1:
> > +        break;
> > +    case 2:
> > +        val = bswap16(val);
> > +        break;
> > +    case 4:
> > +        val = bswap32(val);
> > +        break;
> > +    default:
> > +        g_assert_not_reached();
> > +    }
> > +    pci_host_config_write_common(pdev, cfg_addr, limit, val, size);
> > +}
> > +
> > +static uint64_t pnv_phb3_config_read(PnvPHB3 *phb, unsigned off,
> > +                                     unsigned size)
> > +{
> > +    uint32_t cfg_addr, limit;
> > +    PCIDevice *pdev;
> > +    uint64_t val;
> > +
> > +    pdev = pnv_phb3_find_cfg_dev(phb);
> > +    if (!pdev) {
> > +        return ~0ull;
> > +    }
> > +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xffc;
> > +    cfg_addr |= off;
> > +    limit = pci_config_size(pdev);
> > +    if (limit <= cfg_addr) {
> > +        /* conventional pci device can be behind pcie-to-pci bridge.
> > +           256 <= addr < 4K has no effects. */
> > +        return ~0ull;
> > +    }
> > +    val = pci_host_config_read_common(pdev, cfg_addr, limit, size);
> > +    switch (size) {
> > +    case 1:
> > +        return val;
> > +    case 2:
> > +        return bswap16(val);
> > +    case 4:
> > +        return bswap32(val);
> > +    default:
> > +        g_assert_not_reached();
> > +    }
> > +}
> > +
> > +static void pnv_phb3_check_m32(PnvPHB3 *phb)
> > +{
> > +    uint64_t base, start, size;
> > +    MemoryRegion *parent;
> > +    PnvPBCQState *pbcq = &phb->pbcq;
> > +
> > +    if (phb->m32_mapped) {
> > +        memory_region_del_subregion(phb->mr_m32.container, &phb->mr_m32);
> > +        phb->m32_mapped = false;
> > +    }
> > +
> > +    /* Disabled ? move on with life ... */
> > +    if (!(phb->regs[PHB_PHB3_CONFIG >> 3] & PHB_PHB3C_M32_EN)) {
> > +        return;
> > +    }
> > +
> > +    /* Grab geometry from registers */
> > +    base = phb->regs[PHB_M32_BASE_ADDR >> 3];
> > +    start = phb->regs[PHB_M32_START_ADDR >> 3];
> > +    size = ~(phb->regs[PHB_M32_BASE_MASK >> 3] | 0xfffc000000000000ull) + 1;
> > +
> > +    /* Check if it matches an enabled MMIO region in the PBCQ */
> > +    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
> > +        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
> > +        parent = &pbcq->mmbar0;
> > +        base -= pbcq->mmio0_base;
> > +    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
> > +        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
> > +        parent = &pbcq->mmbar1;
> > +        base -= pbcq->mmio1_base;
> > +    } else {
> > +        return;
> > +    }
> > +
> > +    /* Create alias */
> > +    memory_region_init_alias(&phb->mr_m32, OBJECT(phb), "phb3-m32",
> > +                             &phb->pci_mmio, start, size);
> > +    memory_region_add_subregion(parent, base, &phb->mr_m32);
> > +    phb->m32_mapped = true;
> > +}
> > +
> > +static void pnv_phb3_check_m64(PnvPHB3 *phb, uint32_t index)
> > +{
> > +    uint64_t base, start, size, m64;
> > +    MemoryRegion *parent;
> > +    PnvPBCQState *pbcq = &phb->pbcq;
> > +
> > +    if (phb->m64_mapped[index]) {
> > +        /* Should we destroy it in RCU friendly way... ? */
> > +        memory_region_del_subregion(phb->mr_m64[index].container,
> > +                                    &phb->mr_m64[index]);
> > +        phb->m64_mapped[index] = false;
> > +    }
> > +
> > +    /* Get table entry */
> > +    m64 = phb->ioda_M64BT[index];
> > +
> > +    /* Disabled ? move on with life ... */
> > +    if (!(m64 & IODA2_M64BT_ENABLE)) {
> > +        return;
> > +    }
> > +
> > +    /* Grab geometry from registers */
> > +    base = GETFIELD(IODA2_M64BT_BASE, m64) << 20;
> > +    if (m64 & IODA2_M64BT_SINGLE_PE) {
> > +        base &= ~0x1ffffffull;
> > +    }
> > +    size = GETFIELD(IODA2_M64BT_MASK, m64) << 20;
> > +    size |= 0xfffc000000000000ull;
> > +    size = ~size + 1;
> > +    start = base | (phb->regs[PHB_M64_UPPER_BITS >> 3]);
> > +
> > +    /* Check if it matches an enabled MMIO region in the PBCQ */
> > +    if (pbcq->mmio0_mapped && base >= pbcq->mmio0_base &&
> > +        (base + size) <= (pbcq->mmio0_base + pbcq->mmio0_size)) {
> > +        parent = &pbcq->mmbar0;
> > +        base -= pbcq->mmio0_base;
> > +    } else if (pbcq->mmio1_mapped && base >= pbcq->mmio1_base &&
> > +        (base + size) <= (pbcq->mmio1_base + pbcq->mmio1_size)) {
> > +        parent = &pbcq->mmbar1;
> > +        base -= pbcq->mmio1_base;
> > +    } else {
> > +        return;
> > +    }
> > +
> > +    /* Create alias */
> > +    memory_region_init_alias(&phb->mr_m64[index], OBJECT(phb), "phb3-m64",
> > +                             &phb->pci_mmio, start, size);
> > +    memory_region_add_subregion(parent, base, &phb->mr_m64[index]);
> > +    phb->m64_mapped[index] = true;
> > +}
> > +
> > +static void pnv_phb3_check_all_m64s(PnvPHB3 *phb)
> > +{
> > +    uint64_t i;
> > +
> > +    for (i = 0; i < PNV_PHB3_NUM_M64; i++) {
> > +        pnv_phb3_check_m64(phb, i);
> > +    }
> > +}
> > +
> > +static void pnv_phb3_lxivt_write(PnvPHB3 *phb, unsigned idx, uint64_t val)
> > +{
> > +    ICSState *ics = &phb->lsis;
> > +    uint8_t server, prio;
> > +
> > +    phb->ioda_LXIVT[idx] = val & (IODA2_LXIVT_SERVER |
> > +                                  IODA2_LXIVT_PRIORITY |
> > +                                  IODA2_LXIVT_NODE_ID);
> > +    server = GETFIELD(IODA2_LXIVT_SERVER, val);
> > +    prio = GETFIELD(IODA2_LXIVT_PRIORITY, val);
> > +
> > +    /*
> > +     * The low order 2 bits are the link pointer (Type II interrupts).
> > +     * Shift back to get a valid IRQ server.
> > +     */
> > +    server >>= 2;
> > +
> > +    ics_simple_write_xive(ics, idx, server, prio, prio);
> > +}
> > +
> > +static uint64_t *pnv_phb3_ioda_access(PnvPHB3 *phb,
> > +                                      unsigned *out_table, unsigned *out_idx)
> > +{
> > +    uint64_t adreg = phb->regs[PHB_IODA_ADDR >> 3];
> > +    unsigned int index = GETFIELD(PHB_IODA_AD_TADR, adreg);
> > +    unsigned int table = GETFIELD(PHB_IODA_AD_TSEL, adreg);
> > +    unsigned int mask;
> > +    uint64_t *tptr = NULL;
> > +
> > +    switch (table) {
> > +    case IODA2_TBL_LIST:
> > +        tptr = phb->ioda_LIST;
> > +        mask = 7;
> > +        break;
> > +    case IODA2_TBL_LXIVT:
> > +        tptr = phb->ioda_LXIVT;
> > +        mask = 7;
> > +        break;
> > +    case IODA2_TBL_IVC_CAM:
> > +    case IODA2_TBL_RBA:
> > +        mask = 31;
> > +        break;
> > +    case IODA2_TBL_RCAM:
> > +        mask = 63;
> > +        break;
> > +    case IODA2_TBL_MRT:
> > +        mask = 7;
> > +        break;
> > +    case IODA2_TBL_PESTA:
> > +    case IODA2_TBL_PESTB:
> > +        mask = 255;
> > +        break;
> > +    case IODA2_TBL_TVT:
> > +        tptr = phb->ioda_TVT;
> > +        mask = 511;
> > +        break;
> > +    case IODA2_TBL_TCAM:
> > +    case IODA2_TBL_TDR:
> > +        mask = 63;
> > +        break;
> > +    case IODA2_TBL_M64BT:
> > +        tptr = phb->ioda_M64BT;
> > +        mask = 15;
> > +        break;
> > +    case IODA2_TBL_M32DT:
> > +        tptr = phb->ioda_MDT;
> > +        mask = 255;
> > +        break;
> > +    case IODA2_TBL_PEEV:
> > +        tptr = phb->ioda_PEEV;
> > +        mask = 3;
> > +        break;
> > +    default:
> > +        return NULL;
> > +    }
> > +    index &= mask;
> > +    if (out_idx) {
> > +        *out_idx = index;
> > +    }
> > +    if (out_table) {
> > +        *out_table = table;
> > +    }
> > +    if (adreg & PHB_IODA_AD_AUTOINC) {
> > +        index = (index + 1) & mask;
> > +        adreg = SETFIELD(PHB_IODA_AD_TADR, adreg, index);
> > +    }
> > +    if (tptr) {
> > +        tptr += index;
> > +    }
> > +    phb->regs[PHB_IODA_ADDR >> 3] = adreg;
> > +    return tptr;
> > +}
> > +
> > +static uint64_t pnv_phb3_ioda_read(PnvPHB3 *phb)
> > +{
> > +        unsigned table;
> > +        uint64_t *tptr;
> > +
> > +        tptr = pnv_phb3_ioda_access(phb, &table, NULL);
> > +        if (!tptr) {
> > +            /* Return 0 on unsupported tables, not ff's */
> > +            return 0;
> > +        }
> > +        return *tptr;
> > +}
> > +
> > +static void pnv_phb3_ioda_write(PnvPHB3 *phb, uint64_t val)
> > +{
> > +        unsigned table, idx;
> > +        uint64_t *tptr;
> > +
> > +        tptr = pnv_phb3_ioda_access(phb, &table, &idx);
> > +        if (!tptr) {
> > +            return;
> > +        }
> > +
> > +        /* Handle side effects */
> > +        switch (table) {
> > +        case IODA2_TBL_LXIVT:
> > +            pnv_phb3_lxivt_write(phb, idx, val);
> > +            break;
> > +        case IODA2_TBL_M64BT:
> > +            *tptr = val;
> > +            pnv_phb3_check_m64(phb, idx);
> > +            break;
> > +        default:
> > +            *tptr = val;
> > +        }
> > +}
> > +
> > +/* This is called whenever the PHB LSI, MSI source ID register or
> > + * the PBCQ irq filters are written.
> > + */
> > +void pnv_phb3_remap_irqs(PnvPHB3 *phb)
> > +{
> > +    ICSState *ics = &phb->lsis;
> > +    uint32_t local, global, count, mask, comp;
> > +    uint64_t baren;
> > +    PnvPBCQState *pbcq = &phb->pbcq;
> > +
> > +    /* First check if we are enabled. Unlike real HW we don't separate TX and RX
> > +     * so we enable if both are set
> > +     */
> > +    baren = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
> > +    if (!(baren & PBCQ_NEST_BAR_EN_IRSN_RX) ||
> > +        !(baren & PBCQ_NEST_BAR_EN_IRSN_TX)) {
> > +        ics->offset = 0;
> > +        return;
> > +    }
> > +
> > +    /* Grab local LSI source ID */
> > +    local = GETFIELD(PHB_LSI_SRC_ID, phb->regs[PHB_LSI_SOURCE_ID >> 3]) << 3;
> > +
> > +    /* Grab global one and compare */
> > +    global = GETFIELD(PBCQ_NEST_LSI_SRC,
> > +                      pbcq->nest_regs[PBCQ_NEST_LSI_SRC_ID]) << 3;
> > +    if (global != local) {
> > +        /* This happens during initialization, let's come back when we
> > +         * are properly configured
> > +         */
> > +        ics->offset = 0;
> > +        return;
> > +    }
> > +
> > +    /* Get the base on the powerbus */
> > +    comp = GETFIELD(PBCQ_NEST_IRSN_COMP,
> > +                    pbcq->nest_regs[PBCQ_NEST_IRSN_COMPARE]);
> > +    mask = GETFIELD(PBCQ_NEST_IRSN_COMP,
> > +                    pbcq->nest_regs[PBCQ_NEST_IRSN_MASK]);
> > +    count = ((~mask) + 1) & 0x7ffff;
> > +    phb->total_irq = count;
> > +
> > +    /* Sanity checks */
> > +    if ((global + PNV_PHB3_NUM_LSI) > count) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "LSIs out of reach: LSI base=%d total irq=%d",
> > +                      global, count);
> > +    }
> > +
> > +    if (count > 2048) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "More interrupts than supported: %d", count);
> > +    }
> > +
> > +    if ((comp & mask) != comp) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "IRQ compare bits not in mask: comp=0x%x mask=0x%x",
> > +                      comp, mask);
> > +        comp &= mask;
> > +    }
> > +    /* Setup LSI offset */
> > +    ics->offset = comp + global;
> > +
> > +    /* Setup MSI offset */
> > +    pnv_phb3_msi_update_config(&phb->msis, comp, count - PNV_PHB3_NUM_LSI);
> > +}
> > +
> > +static void pnv_phb3_lsi_src_id_write(PnvPHB3 *phb, uint64_t val)
> > +{
> > +    /* Sanitize content */
> > +    val &= PHB_LSI_SRC_ID;
> > +    phb->regs[PHB_LSI_SOURCE_ID >> 3] = val;
> > +    pnv_phb3_remap_irqs(phb);
> > +}
> > +
> > +static void pnv_phb3_rtc_invalidate(PnvPHB3 *phb, uint64_t val)
> > +{
> > +    PnvPhb3DMASpace *ds;
> > +
> > +    /* Always invalidate all for now ... */
> > +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> > +        ds->pe_num = PHB_INVALID_PE;
> > +    }
> > +}
> > +
> > +
> > +static void pnv_phb3_update_msi_regions(PnvPhb3DMASpace *ds)
> > +{
> > +    uint64_t cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
> > +
> > +    if (cfg & PHB_PHB3C_32BIT_MSI_EN) {
> > +        if (!ds->msi32_mapped) {
> > +            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
> > +                                        0xffff0000, &ds->msi32_mr);
> > +            ds->msi32_mapped = true;
> > +        }
> > +    } else {
> > +        if (ds->msi32_mapped) {
> > +            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
> > +                                        &ds->msi32_mr);
> > +            ds->msi32_mapped = false;
> > +        }
> > +    }
> > +
> > +    if (cfg & PHB_PHB3C_64BIT_MSI_EN) {
> > +        if (!ds->msi64_mapped) {
> > +            memory_region_add_subregion(MEMORY_REGION(&ds->dma_mr),
> > +                                        (1ull << 60), &ds->msi64_mr);
> > +            ds->msi64_mapped = true;
> > +        }
> > +    } else {
> > +        if (ds->msi64_mapped) {
> > +            memory_region_del_subregion(MEMORY_REGION(&ds->dma_mr),
> > +                                        &ds->msi64_mr);
> > +            ds->msi64_mapped = false;
> > +        }
> > +    }
> > +}
> > +
> > +static void pnv_phb3_update_all_msi_regions(PnvPHB3 *phb)
> > +{
> > +    PnvPhb3DMASpace *ds;
> > +
> > +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> > +        pnv_phb3_update_msi_regions(ds);
> > +    }
> > +}
> > +
> > +void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size)
> > +{
> > +    PnvPHB3 *phb = opaque;
> > +    bool changed;
> > +
> > +    /* Special case configuration data */
> > +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> > +        pnv_phb3_config_write(phb, off & 0x3, size, val);
> > +        return;
> > +    }
> > +
> > +    /* Other registers are 64-bit only */
> > +    if (size != 8 || off & 0x7) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> > +                      off, size);
> > +        return;
> > +    }
> > +
> > +    /* Handle masking */
> > +    switch (off) {
> > +    case PHB_M64_UPPER_BITS:
> > +        val &= 0xfffc000000000000ull;
> > +        break;
> > +    }
> > +
> > +    /* Record whether it changed */
> > +    changed = phb->regs[off >> 3] != val;
> > +
> > +    /* Store in register cache first */
> > +    phb->regs[off >> 3] = val;
> > +
> > +    /* Handle side effects */
> > +    switch (off) {
> > +    case PHB_PHB3_CONFIG:
> > +        if (changed) {
> > +            pnv_phb3_update_all_msi_regions(phb);
> > +        }
> > +        /* fall through */
> > +    case PHB_M32_BASE_ADDR:
> > +    case PHB_M32_BASE_MASK:
> > +    case PHB_M32_START_ADDR:
> > +        if (changed) {
> > +            pnv_phb3_check_m32(phb);
> > +        }
> > +        break;
> > +    case PHB_M64_UPPER_BITS:
> > +        if (changed) {
> > +            pnv_phb3_check_all_m64s(phb);
> > +        }
> > +        break;
> > +    case PHB_LSI_SOURCE_ID:
> > +        if (changed) {
> > +            pnv_phb3_lsi_src_id_write(phb, val);
> > +        }
> > +        break;
> > +
> > +    /* IODA table accesses */
> > +    case PHB_IODA_DATA0:
> > +        pnv_phb3_ioda_write(phb, val);
> > +        break;
> > +
> > +    /* RTC invalidation */
> > +    case PHB_RTC_INVALIDATE:
> > +        pnv_phb3_rtc_invalidate(phb, val);
> > +        break;
> > +
> > +    /* FFI request */
> > +    case PHB_FFI_REQUEST:
> > +        pnv_phb3_msi_ffi(&phb->msis, val);
> > +        break;
> > +
> > +    /* Silent simple writes */
> > +    case PHB_CONFIG_ADDRESS:
> > +    case PHB_IODA_ADDR:
> > +    case PHB_TCE_KILL:
> > +    case PHB_TCE_SPEC_CTL:
> > +    case PHB_PEST_BAR:
> > +    case PHB_PELTV_BAR:
> > +    case PHB_RTT_BAR:
> > +    case PHB_RBA_BAR:
> > +    case PHB_IVT_BAR:
> > +    case PHB_FFI_LOCK:
> > +        break;
> > +
> > +    /* Noise on anything else */
> > +    default:
> > +        qemu_log_mask(LOG_UNIMP, "reg_write 0x%"PRIx64"=%"PRIx64, off, val);
> > +    }
> > +}
> > +
> > +uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size)
> > +{
> > +    PnvPHB3 *phb = opaque;
> > +    uint64_t val;
> > +
> > +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> > +        return pnv_phb3_config_read(phb, off & 0x3, size);
> > +    }
> > +
> > +    /* Other registers are 64-bit only */
> > +    if (size != 8 || off & 0x7) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> > +                      off, size);
> > +        return ~0ull;
> > +    }
> > +
> > +    /* Default read from cache */
> > +    val = phb->regs[off >> 3];
> > +
> > +    switch (off) {
> > +    /* Simulate venice DD2.0 */
> > +    case PHB_VERSION:
> > +        return 0x000000a300000005ull;
> > +
> > +    /* IODA table accesses */
> > +    case PHB_IODA_DATA0:
> > +        return pnv_phb3_ioda_read(phb);
> > +
> > +    /* Link training always appears trained */
> > +    case PHB_PCIE_DLP_TRAIN_CTL:
> > +        return PHB_PCIE_DLP_INBAND_PRESENCE | PHB_PCIE_DLP_TC_DL_LINKACT;
> > +
> > +    /* FFI Lock */
> > +    case PHB_FFI_LOCK:
> > +        /* Set lock and return previous value */
> > +        phb->regs[off >> 3] |= PHB_FFI_LOCK_STATE;
> > +        return val;
> > +
> > +    /* Silent simple reads */
> > +    case PHB_PHB3_CONFIG:
> > +    case PHB_M32_BASE_ADDR:
> > +    case PHB_M32_BASE_MASK:
> > +    case PHB_M32_START_ADDR:
> > +    case PHB_CONFIG_ADDRESS:
> > +    case PHB_IODA_ADDR:
> > +    case PHB_RTC_INVALIDATE:
> > +    case PHB_TCE_KILL:
> > +    case PHB_TCE_SPEC_CTL:
> > +    case PHB_PEST_BAR:
> > +    case PHB_PELTV_BAR:
> > +    case PHB_RTT_BAR:
> > +    case PHB_RBA_BAR:
> > +    case PHB_IVT_BAR:
> > +    case PHB_M64_UPPER_BITS:
> > +        break;
> > +
> > +    /* Noise on anything else */
> > +    default:
> > +        qemu_log_mask(LOG_UNIMP, "reg_read 0x%"PRIx64"=%"PRIx64, off, val);
> > +    }
> > +    return val;
> > +}
> > +
> > +static const MemoryRegionOps pnv_phb3_reg_ops = {
> > +    .read = pnv_phb3_reg_read,
> > +    .write = pnv_phb3_reg_write,
> > +    .valid.min_access_size = 1,
> > +    .valid.max_access_size = 8,
> > +    .impl.min_access_size = 1,
> > +    .impl.max_access_size = 8,
> > +    .endianness = DEVICE_BIG_ENDIAN,
> > +};
> > +
> > +static int pnv_phb3_map_irq(PCIDevice *pci_dev, int irq_num)
> > +{
> > +    /* Check that out properly ... */
> > +    return irq_num & 3;
> > +}
> > +
> > +static void pnv_phb3_set_irq(void *opaque, int irq_num, int level)
> > +{
> > +    PnvPHB3 *phb = opaque;
> > +
> > +    /* LSI only ... */
> > +    if (irq_num > 3) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "Unknown IRQ to set %d", irq_num);
> > +    }
> > +    qemu_set_irq(phb->lsis.qirqs[irq_num], level);
> > +}
> > +
> > +static bool pnv_phb3_resolve_pe(PnvPhb3DMASpace *ds)
> > +{
> > +    uint64_t rtt, addr;
> > +    uint16_t rte;
> > +    int bus_num;
> > +
> > +    /* Already resolved ? */
> > +    if (ds->pe_num != PHB_INVALID_PE) {
> > +        return true;
> > +    }
> > +
> > +    /* We need to lookup the RTT */
> > +    rtt = ds->phb->regs[PHB_RTT_BAR >> 3];
> > +    if (!(rtt & PHB_RBA_BAR_ENABLE)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "DMA with RTT BAR disabled !");
> > +        /* Set error bits ? fence ? ... */
> > +        return false;
> > +    }
> > +
> > +    /* Read RTE */
> > +    bus_num = pci_bus_num(ds->bus);
> > +    addr = rtt & PHB_RTT_BASE_ADDRESS_MASK;
> > +    addr += 2 * ((bus_num << 8) | ds->devfn);
> > +    if (dma_memory_read(&address_space_memory, addr, &rte, sizeof(rte))) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read RTT entry at 0x%"PRIx64,
> > +                      addr);
> > +        /* Set error bits ? fence ? ... */
> > +        return false;
> > +    }
> > +    rte = be16_to_cpu(rte);
> > +
> > +    /* Fail upon reading of invalid PE# */
> > +    if (rte >= PNV_PHB3_NUM_PE) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "RTE for RID 0x%x invalid (%04x)",
> > +                      ds->devfn, rte);
> > +        /* Set error bits ? fence ? ... */
> > +        return false;
> > +    }
> > +    ds->pe_num = rte;
> > +    return true;
> > +}
> > +
> > +static void pnv_phb3_translate_tve(PnvPhb3DMASpace *ds, hwaddr addr,
> > +                                   bool is_write, uint64_t tve,
> > +                                   IOMMUTLBEntry *tlb)
> > +{
> > +    uint64_t tta = GETFIELD(IODA2_TVT_TABLE_ADDR, tve);
> > +    int32_t  lev = GETFIELD(IODA2_TVT_NUM_LEVELS, tve);
> > +    uint32_t tts = GETFIELD(IODA2_TVT_TCE_TABLE_SIZE, tve);
> > +    uint32_t tps = GETFIELD(IODA2_TVT_IO_PSIZE, tve);
> > +
> > +    /* Invalid levels */
> > +    if (lev > 4) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "Invalid #levels in TVE %d", lev);
> > +        return;
> > +    }
> > +
> > +    /* IO Page Size of 0 means untranslated, else use TCEs */
> > +    if (tps == 0) {
> > +        /* We only support non-translate in top window
> > +         * XXX FIX THAT, Venice/Murano support it on bottom window
> > +         * above 4G and Naples suports it on everything
> > +         */
> > +        if (!(tve & PPC_BIT(51))) {
> > +            qemu_log_mask(LOG_GUEST_ERROR,
> > +                          "xlate for invalid non-translate TVE");
> > +            return;
> > +        }
> > +        /* XXX Handle boundaries */
> > +
> > +        /* XXX Use 4k pages like q35 ... for now */
> > +        tlb->iova = addr & 0xfffffffffffff000ull;
> > +        tlb->translated_addr = addr & 0x0003fffffffff000ull;
> > +        tlb->addr_mask = 0xfffull;
> > +        tlb->perm = IOMMU_RW;
> > +    } else {
> > +        uint32_t tce_shift, tbl_shift, sh;
> > +        uint64_t base, taddr, tce, tce_mask;
> > +
> > +        /* TVE disabled ? */
> > +        if (tts == 0) {
> > +            qemu_log_mask(LOG_GUEST_ERROR, "xlate for invalid translated TVE");
> > +            return;
> > +        }
> > +
> > +        /* Address bits per bottom level TCE entry */
> > +        tce_shift = tps + 11;
> > +
> > +        /* Address bits per table level */
> > +        tbl_shift = tts + 8;
> > +
> > +        /* Top level table base address */
> > +        base = tta << 12;
> > +
> > +        /* Total shift to first level */
> > +        sh = tbl_shift * lev + tce_shift;
> > +
> > +        /* XXX Multi-level untested */
> > +        while ((lev--) >= 0) {
> > +            /* Grab the TCE address */
> > +            taddr = base | (((addr >> sh) & ((1ul << tbl_shift) - 1)) << 3);
> > +            if (dma_memory_read(&address_space_memory, taddr, &tce,
> > +                                sizeof(tce))) {
> > +                qemu_log_mask(LOG_GUEST_ERROR,
> > +                              "Failed to read TCE at 0x%"PRIx64, taddr);
> > +                return;
> > +            }
> > +            tce = be64_to_cpu(tce);
> > +
> > +            /* Check permission for indirect TCE */
> > +            if ((lev >= 0) && !(tce & 3)) {
> > +                qemu_log_mask(LOG_GUEST_ERROR,
> > +                              "Invalid indirect TCE at 0x%"PRIx64, taddr);
> > +                qemu_log_mask(LOG_GUEST_ERROR,
> > +                              " xlate %"PRIx64":%c TVE=%"PRIx64,
> > +                              addr, is_write ? 'W' : 'R', tve);
> > +                qemu_log_mask(LOG_GUEST_ERROR,
> > +                              " tta=%"PRIx64" lev=%d tts=%d tps=%d",
> > +                              tta, lev, tts, tps);
> > +                return;
> > +            }
> > +            sh -= tbl_shift;
> > +            base = tce & ~0xfffull;
> > +        }
> > +
> > +        /* We exit the loop with TCE being the final TCE */
> > +        tce_mask = ~((1ull << tce_shift) - 1);
> > +        tlb->iova = addr & tce_mask;
> > +        tlb->translated_addr = tce & tce_mask;
> > +        tlb->addr_mask = ~tce_mask;
> > +        tlb->perm = tce & 3;
> > +        if ((is_write & !(tce & 2)) || ((!is_write) && !(tce & 1))) {
> > +            qemu_log_mask(LOG_GUEST_ERROR, "TCE access fault at 0x%"PRIx64,
> > +                          taddr);
> > +            qemu_log_mask(LOG_GUEST_ERROR, " xlate %"PRIx64":%c TVE=%"PRIx64,
> > +                          addr, is_write ? 'W' : 'R', tve);
> > +            qemu_log_mask(LOG_GUEST_ERROR,
> > +                          " tta=%"PRIx64" lev=%d tts=%d tps=%d",
> > +                          tta, lev, tts, tps);
> > +        }
> > +    }
> > +}
> > +
> > +static IOMMUTLBEntry pnv_phb3_translate_iommu(IOMMUMemoryRegion *iommu,
> > +                                              hwaddr addr,
> > +                                              IOMMUAccessFlags flag,
> > +                                              int iommu_idx)
> > +{
> > +    PnvPhb3DMASpace *ds = container_of(iommu, PnvPhb3DMASpace, dma_mr);
> > +    int tve_sel;
> > +    uint64_t tve, cfg;
> > +    IOMMUTLBEntry ret = {
> > +        .target_as = &address_space_memory,
> > +        .iova = addr,
> > +        .translated_addr = 0,
> > +        .addr_mask = ~(hwaddr)0,
> > +        .perm = IOMMU_NONE,
> > +    };
> > +
> > +    /* Resolve PE# */
> > +    if (!pnv_phb3_resolve_pe(ds)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> > +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> > +        return ret;
> > +    }
> > +
> > +    /* Check top bits */
> > +    switch (addr >> 60) {
> > +    case 00:
> > +        /* DMA or 32-bit MSI ? */
> > +        cfg = ds->phb->regs[PHB_PHB3_CONFIG >> 3];
> > +        if ((cfg & PHB_PHB3C_32BIT_MSI_EN) &&
> > +            ((addr & 0xffffffffffff0000ull) == 0xffff0000ull)) {
> > +            qemu_log_mask(LOG_GUEST_ERROR, "xlate on 32-bit MSI region");
> > +            return ret;
> > +        }
> > +        /* Choose TVE XXX Use PHB3 Control Register */
> > +        tve_sel = (addr >> 59) & 1;
> > +        tve = ds->phb->ioda_TVT[ds->pe_num * 2 + tve_sel];
> > +        pnv_phb3_translate_tve(ds, addr, flag & IOMMU_WO, tve, &ret);
> > +        break;
> > +    case 01:
> > +        qemu_log_mask(LOG_GUEST_ERROR, "xlate on 64-bit MSI region");
> > +        break;
> > +    default:
> > +        qemu_log_mask(LOG_GUEST_ERROR, "xlate on unsupported address 0x%"PRIx64,
> > +                      addr);
> > +    }
> > +    return ret;
> > +}
> > +
> > +#define TYPE_PNV_PHB3_IOMMU_MEMORY_REGION "pnv-phb3-iommu-memory-region"
> > +#define PNV_PHB3_IOMMU_MEMORY_REGION(obj) \
> > +    OBJECT_CHECK(IOMMUMemoryRegion, (obj), TYPE_PNV_PHB3_IOMMU_MEMORY_REGION)
> > +
> > +static void pnv_phb3_iommu_memory_region_class_init(ObjectClass *klass,
> > +                                                    void *data)
> > +{
> > +    IOMMUMemoryRegionClass *imrc = IOMMU_MEMORY_REGION_CLASS(klass);
> > +
> > +    imrc->translate = pnv_phb3_translate_iommu;
> > +}
> > +
> > +static const TypeInfo pnv_phb3_iommu_memory_region_info = {
> > +    .parent = TYPE_IOMMU_MEMORY_REGION,
> > +    .name = TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
> > +    .class_init = pnv_phb3_iommu_memory_region_class_init,
> > +};
> > +
> > +/*
> > + * MSI/MSIX memory region implementation.
> > + * The handler handles both MSI and MSIX.
> > + */
> > +static void pnv_phb3_msi_write(void *opaque, hwaddr addr,
> > +                               uint64_t data, unsigned size)
> > +{
> > +    PnvPhb3DMASpace *ds = opaque;
> > +
> > +    /* Resolve PE# */
> > +    if (!pnv_phb3_resolve_pe(ds)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> > +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> > +        return;
> > +    }
> > +
> > +    pnv_phb3_msi_send(&ds->phb->msis, addr, data, ds->pe_num);
> > +}
> > +
> > +static const MemoryRegionOps pnv_phb3_msi_ops = {
> > +    /* There is no .read as the read result is undefined by PCI spec */
> > +    .read = NULL,
> > +    .write = pnv_phb3_msi_write,
> > +    .endianness = DEVICE_LITTLE_ENDIAN
> > +};
> > +
> > +static AddressSpace *pnv_phb3_dma_iommu(PCIBus *bus, void *opaque, int devfn)
> > +{
> > +    PnvPHB3 *phb = opaque;
> > +    PnvPhb3DMASpace *ds;
> > +
> > +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> > +        if (ds->bus == bus && ds->devfn == devfn) {
> > +            break;
> > +        }
> > +    }
> > +
> > +    if (ds == NULL) {
> > +        ds = g_malloc0(sizeof(PnvPhb3DMASpace));
> > +        ds->bus = bus;
> > +        ds->devfn = devfn;
> > +        ds->pe_num = PHB_INVALID_PE;
> > +        ds->phb = phb;
> > +        memory_region_init_iommu(&ds->dma_mr, sizeof(ds->dma_mr),
> > +                                 TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
> > +                                 OBJECT(phb), "phb3_iommu", UINT64_MAX);
> > +        address_space_init(&ds->dma_as, MEMORY_REGION(&ds->dma_mr),
> > +                           "phb3_iommu");
> > +        memory_region_init_io(&ds->msi32_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> > +                              ds, "msi32", 0x10000);
> > +        memory_region_init_io(&ds->msi64_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> > +                              ds, "msi64", 0x100000);
> > +        pnv_phb3_update_msi_regions(ds);
> > +
> > +        QLIST_INSERT_HEAD(&phb->dma_spaces, ds, list);
> > +    }
> > +    return &ds->dma_as;
> > +}
> > +
> > +static void pnv_phb3_instance_init(Object *obj)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(obj);
> > +
> > +    /* Create LSI source */
> > +    object_initialize(&phb->lsis, sizeof(phb->lsis), TYPE_ICS_SIMPLE);
> > +    object_property_add_child(obj, "ics-phb-lsi", OBJECT(&phb->lsis), NULL);
> > +
> > +    /* Default init ... will be fixed by HW inits */
> > +    phb->lsis.offset = 0;
> > +
> > +    /* Create MSI source */
> > +    object_initialize(&phb->msis, sizeof(phb->msis), TYPE_PHB3_MSI);
> > +    object_property_add_const_link(OBJECT(&phb->msis), "phb", obj,
> > +                                   &error_abort);
> > +    object_property_add_child(obj, "ics-phb-msi", OBJECT(&phb->msis), NULL);
> > +
> > +    /* Create PBCQ */
> > +    object_initialize(&phb->pbcq, sizeof(phb->pbcq), TYPE_PNV_PBCQ);
> > +    object_property_add_const_link(OBJECT(&phb->pbcq), "phb", obj,
> > +                                   &error_abort);
> > +    object_property_add_child(obj, "pbcq", OBJECT(&phb->pbcq), NULL);
> > +
> > +    QLIST_INIT(&phb->dma_spaces);
> > +}
> > +
> > +/*
> > + * This could be done under pnv_pbcq_realize
> > + */
> > +static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
> > +{
> > +    PCIHostState *pcih = PCI_HOST_BRIDGE(phb);
> > +    PCIDevice *brdev;
> > +    PCIDevice *pdev;
> > +    PCIBus *parent;
> > +    uint8_t chassis = phb->chip_id * 4 + phb->phb_id;
> > +    uint8_t chassis_nr = 128;
> > +
> > +    /* Add root complex */
> > +    pdev = pci_create(pcih->bus, 0, TYPE_PNV_PHB3_RC);
> > +    object_property_add_child(OBJECT(phb), "phb3-rc", OBJECT(pdev), NULL);
> > +    qdev_prop_set_uint8(DEVICE(pdev), "chassis", chassis);
> > +    qdev_prop_set_uint16(DEVICE(pdev), "slot", 1);
> > +    qdev_init_nofail(DEVICE(pdev));
> > +
> > +    /* Setup bus for that chip */
> > +    parent = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
> > +
> > +    brdev = pci_create(parent, 0, "pci-bridge");
> > +    object_property_add_child(OBJECT(parent), "pci-bridge", OBJECT(brdev),
> > +                              NULL);
> > +    qdev_prop_set_uint8(DEVICE(brdev), PCI_BRIDGE_DEV_PROP_CHASSIS_NR,
> > +                        chassis_nr);
> > +    qdev_init_nofail(DEVICE(brdev));
> > +}
> > +
> > +static void pnv_phb3_realize(DeviceState *dev, Error **errp)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(dev);
> > +    PCIHostState *pci = PCI_HOST_BRIDGE(dev);
> > +    Object *xics = OBJECT(qdev_get_machine());
> > +    Error *local_err = NULL;
> > +    int i;
> > +
> > +    memory_region_init(&phb->pci_mmio, OBJECT(phb), "pci-mmio",
> > +                       PCI_MMIO_TOTAL_SIZE);
> > +
> > +    /* PHB3 doesn't support IO space. However, qemu gets very upset if
> > +     * we don't have an IO region to anchor IO BARs onto so we just
> > +     * initialize one which we never hook up to anything
> > +     */
> > +    memory_region_init(&phb->pci_io, OBJECT(phb), "pci-io", 0x10000);
> > +
> > +    memory_region_init_io(&phb->mr_regs, OBJECT(phb), &pnv_phb3_reg_ops, phb,
> > +                          "phb3-regs", 0x1000);
> > +
> > +    object_property_set_int(OBJECT(&phb->lsis), PNV_PHB3_NUM_LSI, "nr-irqs",
> > +                            &error_abort);
> > +    object_property_add_const_link(OBJECT(&phb->lsis), "xics", xics,
> > +                                   &error_abort);
> > +    object_property_set_bool(OBJECT(&phb->lsis), true, "realized", &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    for (i = 0; i < PNV_PHB3_NUM_LSI; i++) {
> > +        ics_set_irq_type(&phb->lsis, i, true);
> > +    }
> > +
> > +    object_property_add_const_link(OBJECT(&phb->msis), "xics", xics,
> > +                                   &error_abort);
> > +    object_property_set_int(OBJECT(&phb->msis), PHB3_MAX_MSI, "nr-irqs",
> > +                            &error_abort);
> > +    object_property_set_bool(OBJECT(&phb->msis), true, "realized", &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    object_property_set_bool(OBJECT(&phb->pbcq), true, "realized", &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    pci->bus = pci_register_root_bus(dev, "phb3-root-bus",
> > +                                pnv_phb3_set_irq, pnv_phb3_map_irq, phb,
> > +                                &phb->pci_mmio, &phb->pci_io,
> > +                                0, 4, TYPE_PNV_PHB3_ROOT_BUS);
> > +    pci_setup_iommu(pci->bus, pnv_phb3_dma_iommu, phb);
> > +
> > +    /* Setup the PCI busses */
> > +    pnv_phb3_pci_create(phb, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +}
> > +
> > +void pnv_phb3_update_regions(PnvPHB3 *phb)
> > +{
> > +    PnvPBCQState *pbcq = &phb->pbcq;
> > +
> > +    /* Unmap first always */
> > +    if (phb->regs_mapped) {
> > +        memory_region_del_subregion(&pbcq->phbbar, &phb->mr_regs);
> > +        phb->regs_mapped = false;
> > +    }
> > +
> > +    /* Map registers if enabled */
> > +    if (pbcq->phb_mapped) {
> > +        /* XXX We should use the PHB BAR 2 register but we don't ... */
> > +        memory_region_add_subregion(&pbcq->phbbar, 0, &phb->mr_regs);
> > +        phb->regs_mapped = true;
> > +    }
> > +
> > +    /* Check/update m32 */
> > +    if (phb->m32_mapped) {
> > +        pnv_phb3_check_m32(phb);
> > +    }
> > +}
> > +
> > +static const char *pnv_phb3_root_bus_path(PCIHostState *host_bridge,
> > +                                          PCIBus *rootbus)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(host_bridge);
> > +
> > +    snprintf(phb->bus_path, sizeof(phb->bus_path), "00%02x:%02x",
> > +             phb->chip_id, phb->phb_id);
> > +    return phb->bus_path;
> > +}
> > +
> > +static void pnv_phb3_get_phb_id(Object *obj, Visitor *v, const char *name,
> > +                              void *opaque, Error **errp)
> > +{
> > +    Property *prop = opaque;
> > +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> > +
> > +    visit_type_uint32(v, name, ptr, errp);
> > +}
> > +
> > +static void pnv_phb3_set_phb_id(Object *obj, Visitor *v, const char *name,
> > +                              void *opaque, Error **errp)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(obj);
> > +    uint32_t phb_id;
> > +    Error *local_err = NULL;
> > +
> > +    visit_type_uint32(v, name, &phb_id, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    /*
> > +     * Limit to a maximum of 6 PHBs per chip
> > +     */
> > +    if (phb_id >= PNV8_CHIP_PHB3_MAX) {
> > +        error_setg(errp, "invalid PHB index: '%d'", phb_id);
> > +        return;
> > +    }
> > +
> > +    phb->phb_id = phb_id;
> > +}
> > +
> > +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
> > +    .name = "irq",
> > +    .get = pnv_phb3_get_phb_id,
> > +    .set = pnv_phb3_set_phb_id,
> > +};
> > +
> > +static void pnv_phb3_get_chip_id(Object *obj, Visitor *v, const char *name,
> > +                              void *opaque, Error **errp)
> > +{
> > +    Property *prop = opaque;
> > +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> > +
> > +    visit_type_uint32(v, name, ptr, errp);
> > +}
> > +
> > +static void pnv_phb3_set_chip_id(Object *obj, Visitor *v, const char *name,
> > +                                 void *opaque, Error **errp)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(obj);
> > +    uint32_t chip_id;
> > +    Error *local_err = NULL;
> > +    PnvChip *chip;
> > +
> > +    visit_type_uint32(v, name, &chip_id, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    /*
> > +     * Check that the chip owning the device is valid.
> > +     */
> > +    chip = pnv_get_chip(PNV_MACHINE(qdev_get_machine()), chip_id);
> > +    if (!chip) {
> > +        error_setg(errp, "chip id '%d' is invalid", chip_id);
> > +        return;
> > +    }
> > +
> > +    phb->chip_id = chip_id;
> > +}
> > +
> > +static const PropertyInfo pnv_phb3_chip_id_propinfo = {
> > +    .name = "irq",
> > +    .get = pnv_phb3_get_chip_id,
> > +    .set = pnv_phb3_set_chip_id,
> > +};
> > +
> > +static Property pnv_phb3_properties[] = {
> > +    DEFINE_PROP("phb-id", PnvPHB3, phb_id, pnv_phb3_phb_id_propinfo,
> > +                uint32_t),
> > +    DEFINE_PROP("chip-id", PnvPHB3, chip_id, pnv_phb3_chip_id_propinfo,
> > +                uint32_t),
> > +    DEFINE_PROP_END_OF_LIST(),
> > +};
> > +
> > +static void pnv_phb3_class_init(ObjectClass *klass, void *data)
> > +{
> > +    PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_CLASS(klass);
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +
> > +    hc->root_bus_path = pnv_phb3_root_bus_path;
> > +    dc->realize = pnv_phb3_realize;
> > +    dc->props = pnv_phb3_properties;
> > +    set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
> > +}
> > +
> > +static const TypeInfo pnv_phb3_type_info = {
> > +    .name          = TYPE_PNV_PHB3,
> > +    .parent        = TYPE_PCI_HOST_BRIDGE,
> > +    .instance_size = sizeof(PnvPHB3),
> > +    .class_init    = pnv_phb3_class_init,
> > +    .instance_init = pnv_phb3_instance_init,
> > +};
> > +
> > +static void pnv_phb3_root_bus_class_init(ObjectClass *klass, void *data)
> > +{
> > +    BusClass *k = BUS_CLASS(klass);
> > +
> > +    k->max_dev = 1;
> > +}
> > +
> > +static const TypeInfo pnv_phb3_root_bus_info = {
> > +    .name = TYPE_PNV_PHB3_ROOT_BUS,
> > +    .parent = TYPE_PCIE_BUS,
> > +    .class_init = pnv_phb3_root_bus_class_init,
> > +};
> > +
> > +static void pnv_phb3_rc_class_init(ObjectClass *klass, void *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
> > +    PCIERootPortClass *rpc = PCIE_ROOT_PORT_CLASS(klass);
> > +
> > +    dc->desc     = "IBM PHB3 PCIE Root Port";
> > +
> > +    k->vendor_id = PCI_VENDOR_ID_IBM;
> > +    k->device_id = 0x03dc;
> > +    k->revision  = 0;
> > +    rpc->exp_offset = 0x48;
> > +    rpc->aer_offset = 0x100;
> > +}
> > +
> > +static const TypeInfo pnv_phb3_rc_info = {
> > +    .name          = TYPE_PNV_PHB3_RC,
> > +    .parent        = TYPE_PCIE_ROOT_PORT,
> > +    .class_init    = pnv_phb3_rc_class_init,
> > +};
> > +
> > +static void pnv_phb3_register_types(void)
> > +{
> > +    type_register_static(&pnv_phb3_rc_info);
> > +    type_register_static(&pnv_phb3_type_info);
> > +    type_register_static(&pnv_phb3_root_bus_info);
> > +    type_register_static(&pnv_phb3_iommu_memory_region_info);
> > +}
> > +
> > +type_init(pnv_phb3_register_types)
> > diff --git a/hw/pci-host/pnv_phb3_msi.c b/hw/pci-host/pnv_phb3_msi.c
> > new file mode 100644
> > index 000000000000..36927da25b61
> > --- /dev/null
> > +++ b/hw/pci-host/pnv_phb3_msi.c
> > @@ -0,0 +1,316 @@
> > +/*
> > + * QEMU PowerPC PowerNV PHB3 model
> > + *
> > + * Copyright (c) 2014-2018, IBM Corporation.
> > + *
> > + * This code is licensed under the GPL version 2 or later. See the
> > + * COPYING file in the top-level directory.
> > + */
> > +#include "qemu/osdep.h"
> > +#include "qemu/log.h"
> > +#include "qapi/error.h"
> > +#include "qemu-common.h"
> > +#include "target/ppc/cpu.h"
> > +#include "hw/pci-host/pnv_phb3_regs.h"
> > +#include "hw/pci-host/pnv_phb3.h"
> > +#include "hw/ppc/pnv.h"
> > +#include "hw/pci/msi.h"
> > +
> > +static uint64_t phb3_msi_ive_addr(PnvPHB3 *phb, int srcno)
> > +{
> > +    uint64_t ivtbar = phb->regs[PHB_IVT_BAR >> 3];
> > +    uint64_t phbctl = phb->regs[PHB_CONTROL >> 3];
> > +
> > +    if (!(ivtbar & PHB_IVT_BAR_ENABLE)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "Failed access to disable IVT BAR !");
> > +        return 0;
> > +    }
> > +
> > +    if (srcno >= (ivtbar & PHB_IVT_LENGTH_MASK)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "MSI out of bounds (%d vs  0x%"PRIx64")",
> > +                      srcno, ivtbar & PHB_IVT_LENGTH_MASK);
> > +        return 0;
> > +    }
> > +
> > +    ivtbar &= PHB_IVT_BASE_ADDRESS_MASK;
> > +
> > +    if (phbctl & PHB_CTRL_IVE_128_BYTES) {
> > +        return ivtbar + 128 * srcno;
> > +    } else {
> > +        return ivtbar + 16 * srcno;
> > +    }
> > +}
> > +
> > +static bool phb3_msi_read_ive(PnvPHB3 *phb, int srcno, uint64_t *out_ive)
> > +{
> > +    uint64_t ive_addr, ive;
> > +
> > +    ive_addr = phb3_msi_ive_addr(phb, srcno);
> > +    if (!ive_addr) {
> > +        return false;
> > +    }
> > +
> > +    if (dma_memory_read(&address_space_memory, ive_addr, &ive, sizeof(ive))) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "Failed to read IVE at 0x%" PRIx64,
> > +                      ive_addr);
> > +        return false;
> > +    }
> > +    *out_ive = be64_to_cpu(ive);
> > +
> > +    return true;
> > +}
> > +
> > +static void phb3_msi_set_p(Phb3MsiState *msi, int srcno, uint8_t gen)
> > +{
> > +    uint64_t ive_addr;
> > +    uint8_t p = 0x01 | (gen << 1);
> > +
> > +    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
> > +    if (!ive_addr) {
> > +        return;
> > +    }
> > +
> > +    if (dma_memory_write(&address_space_memory, ive_addr + 4, &p, 1)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Failed to write IVE (set P) at 0x%" PRIx64, ive_addr);
> > +    }
> > +}
> > +
> > +static void phb3_msi_set_q(Phb3MsiState *msi, int srcno)
> > +{
> > +    uint64_t ive_addr;
> > +    uint8_t q = 0x01;
> > +
> > +    ive_addr = phb3_msi_ive_addr(msi->phb, srcno);
> > +    if (!ive_addr) {
> > +        return;
> > +    }
> > +
> > +    if (dma_memory_write(&address_space_memory, ive_addr + 5, &q, 1)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Failed to write IVE (set Q) at 0x%" PRIx64, ive_addr);
> > +    }
> > +}
> > +
> > +static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
> > +{
> > +    ICSState *ics = ICS_BASE(msi);
> > +    uint64_t ive;
> > +    uint64_t server, prio, pq, gen;
> > +
> > +    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
> > +        return;
> > +    }
> > +
> > +    server = GETFIELD(IODA2_IVT_SERVER, ive);
> > +    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
> > +    pq = GETFIELD(IODA2_IVT_Q, ive);
> > +    if (!ignore_p) {
> > +        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
> > +    }
> > +    gen = GETFIELD(IODA2_IVT_GEN, ive);
> > +
> > +    /*
> > +     * The low order 2 bits are the link pointer (Type II interrupts).
> > +     * Shift back to get a valid IRQ server.
> > +     */
> > +    server >>= 2;
> > +
> > +    switch (pq) {
> > +    case 0: /* 00 */
> > +        if (prio == 0xff) {
> > +            /* Masked, set Q */
> > +            phb3_msi_set_q(msi, srcno);
> > +        } else {
> > +            /* Enabled, set P and send */
> > +            phb3_msi_set_p(msi, srcno, gen);
> > +            icp_irq(ics, server, srcno + ics->offset, prio);
> > +        }
> > +        break;
> > +    case 2: /* 10 */
> > +        /* Already pending, set Q */
> > +        phb3_msi_set_q(msi, srcno);
> > +        break;
> > +    case 1: /* 01 */
> > +    case 3: /* 11 */
> > +    default:
> > +        /* Just drop stuff if Q already set */
> > +        break;
> > +    }
> > +}
> > +
> > +static void phb3_msi_set_irq(void *opaque, int srcno, int val)
> > +{
> > +    Phb3MsiState *msi = PHB3_MSI(opaque);
> > +
> > +    if (val) {
> > +        phb3_msi_try_send(msi, srcno, false);
> > +    }
> > +}
> > +
> > +
> > +void pnv_phb3_msi_send(Phb3MsiState *msi, uint64_t addr, uint16_t data,
> > +                       int32_t dev_pe)
> > +{
> > +    ICSState *ics = ICS_BASE(msi);
> > +    uint64_t ive;
> > +    uint16_t pe;
> > +    uint32_t src = ((addr >> 4) & 0xffff) | (data & 0x1f);
> > +
> > +    if (src >= ics->nr_irqs) {
> > +        qemu_log_mask(LOG_GUEST_ERROR, "MSI %d out of bounds", src);
> > +        return;
> > +    }
> > +    if (dev_pe >= 0) {
> > +        if (!phb3_msi_read_ive(msi->phb, src, &ive)) {
> > +            return;
> > +        }
> > +        pe = GETFIELD(IODA2_IVT_PE, ive);
> > +        if (pe != dev_pe) {
> > +            qemu_log_mask(LOG_GUEST_ERROR,
> > +                          "MSI %d send by PE#%d but assigned to PE#%d",
> > +                          src, dev_pe, pe);
> > +            return;
> > +        }
> > +    }
> > +    qemu_irq_pulse(ics->qirqs[src]);
> > +}
> > +
> > +void pnv_phb3_msi_ffi(Phb3MsiState *msi, uint64_t val)
> > +{
> > +    /* Emit interrupt */
> > +    pnv_phb3_msi_send(msi, val, 0, -1);
> > +
> > +    /* Clear FFI lock */
> > +    msi->phb->regs[PHB_FFI_LOCK >> 3] = 0;
> > +}
> > +
> > +static void phb3_msi_reject(ICSState *ics, uint32_t nr)
> > +{
> > +    Phb3MsiState *msi = PHB3_MSI(ics);
> > +    unsigned int srcno = nr - ics->offset;
> > +    unsigned int idx = srcno >> 6;
> > +    unsigned int bit = 1ull << (srcno & 0x3f);
> > +
> > +    assert(srcno < PHB3_MAX_MSI);
> > +
> > +    msi->rba[idx] |= bit;
> > +    msi->rba_sum |= (1u << idx);
> > +}
> > +
> > +static void phb3_msi_resend(ICSState *ics)
> > +{
> > +    Phb3MsiState *msi = PHB3_MSI(ics);
> > +    unsigned int i, j;
> > +
> > +    if (msi->rba_sum == 0) {
> > +        return;
> > +    }
> > +
> > +    for (i = 0; i < 32; i++) {
> > +        if ((msi->rba_sum & (1u << i)) == 0) {
> > +            continue;
> > +        }
> > +        msi->rba_sum &= ~(1u << i);
> > +        for (j = 0; j < 64; j++) {
> > +            if ((msi->rba[i] & (1ull << j)) == 0) {
> > +                continue;
> > +            }
> > +            msi->rba[i] &= ~(1u << j);
> > +            phb3_msi_try_send(msi, i * 64 + j, true);
> > +        }
> > +    }
> > +}
> > +
> > +static void phb3_msi_reset(DeviceState *dev)
> > +{
> > +    Phb3MsiState *msi = PHB3_MSI(dev);
> > +    ICSStateClass *icsc = ICS_BASE_GET_CLASS(dev);
> > +
> > +    icsc->parent_reset(dev);
> > +
> > +    memset(msi->rba, 0, sizeof(msi->rba));
> > +    msi->rba_sum = 0;
> > +}
> > +
> > +static void phb3_msi_reset_handler(void *dev)
> > +{
> > +    phb3_msi_reset(dev);
> > +}
> > +
> > +void pnv_phb3_msi_update_config(Phb3MsiState *msi, uint32_t base,
> > +                                uint32_t count)
> > +{
> > +    ICSState *ics = ICS_BASE(msi);
> > +
> > +    if (count > PHB3_MAX_MSI) {
> > +        count = PHB3_MAX_MSI;
> > +    }
> > +    ics->nr_irqs = count;
> > +    ics->offset = base;
> > +}
> > +
> > +static void phb3_msi_realize(DeviceState *dev, Error **errp)
> > +{
> > +    Phb3MsiState *msi = PHB3_MSI(dev);
> > +    ICSState *ics = ICS_BASE(msi);
> > +    ICSStateClass *icsc = ICS_BASE_GET_CLASS(ics);
> > +    Object *obj;
> > +    Error *local_err = NULL;
> > +
> > +    icsc->parent_realize(dev, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
> > +    if (!obj) {
> > +        error_propagate(errp, local_err);
> > +        error_prepend(errp, "required link 'phb' not found: ");
> > +        return;
> > +    }
> > +    msi->phb = PNV_PHB3(obj);
> > +
> > +    ics->qirqs = qemu_allocate_irqs(phb3_msi_set_irq, msi, PHB3_MAX_MSI);
> > +
> > +    qemu_register_reset(phb3_msi_reset_handler, dev);
> > +}
> > +
> > +static void phb3_msi_instance_init(Object *obj)
> > +{
> > +    ICSState *ics = ICS_BASE(obj);
> > +
> > +    /* Will be overriden later */
> > +    ics->offset = 0;
> > +}
> > +
> > +static void phb3_msi_class_init(ObjectClass *klass, void *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    ICSStateClass *isc = ICS_BASE_CLASS(klass);
> > +
> > +    device_class_set_parent_realize(dc, phb3_msi_realize,
> > +                                    &isc->parent_realize);
> > +    device_class_set_parent_reset(dc, phb3_msi_reset,
> > +                                  &isc->parent_reset);
> > +
> > +    isc->reject = phb3_msi_reject;
> > +    isc->resend = phb3_msi_resend;
> > +}
> > +
> > +static const TypeInfo phb3_msi_info = {
> > +    .name = TYPE_PHB3_MSI,
> > +    .parent = TYPE_ICS_BASE,
> > +    .instance_size = sizeof(Phb3MsiState),
> > +    .class_init = phb3_msi_class_init,
> > +    .class_size = sizeof(ICSStateClass),
> > +    .instance_init = phb3_msi_instance_init,
> > +};
> > +
> > +static void pnv_phb3_msi_register_types(void)
> > +{
> > +    type_register_static(&phb3_msi_info);
> > +}
> > +
> > +type_init(pnv_phb3_msi_register_types)
> > diff --git a/hw/pci-host/pnv_phb3_pbcq.c b/hw/pci-host/pnv_phb3_pbcq.c
> > new file mode 100644
> > index 000000000000..34df3ac684e4
> > --- /dev/null
> > +++ b/hw/pci-host/pnv_phb3_pbcq.c
> > @@ -0,0 +1,347 @@
> > +/*
> > + * QEMU PowerPC PowerNV PHB3 model
> > + *
> > + * Copyright (c) 2014-2018, IBM Corporation.
> > + *
> > + * This code is licensed under the GPL version 2 or later. See the
> > + * COPYING file in the top-level directory.
> > + */
> > +#include "qemu/osdep.h"
> > +#include "qapi/error.h"
> > +#include "qemu-common.h"
> > +#include "qemu/log.h"
> > +#include "target/ppc/cpu.h"
> > +#include "hw/ppc/fdt.h"
> > +#include "hw/pci-host/pnv_phb3_regs.h"
> > +#include "hw/pci-host/pnv_phb3.h"
> > +#include "hw/ppc/pnv_xscom.h"
> > +#include "hw/pci/pci_bridge.h"
> > +#include "hw/pci/pci_bus.h"
> > +
> > +#include <libfdt.h>
> > +
> > +static uint64_t pnv_pbcq_nest_xscom_read(void *opaque, hwaddr addr,
> > +                                         unsigned size)
> > +{
> > +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> > +    uint32_t offset = addr >> 3;
> > +
> > +    return pbcq->nest_regs[offset];
> > +}
> > +
> > +static uint64_t pnv_pbcq_pci_xscom_read(void *opaque, hwaddr addr,
> > +                                        unsigned size)
> > +{
> > +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> > +    uint32_t offset = addr >> 3;
> > +
> > +    return pbcq->pci_regs[offset];
> > +}
> > +
> > +static uint64_t pnv_pbcq_spci_xscom_read(void *opaque, hwaddr addr,
> > +                                         unsigned size)
> > +{
> > +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> > +    uint32_t offset = addr >> 3;
> > +
> > +    if (offset == PBCQ_SPCI_ASB_DATA) {
> > +        return pnv_phb3_reg_read(pbcq->phb,
> > +                                 pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR], 8);
> > +    }
> > +    return pbcq->spci_regs[offset];
> > +}
> > +
> > +static void pnv_pbcq_update_map(PnvPBCQState *pbcq)
> > +{
> > +    uint64_t bar_en = pbcq->nest_regs[PBCQ_NEST_BAR_EN];
> > +    uint64_t bar, mask, size;
> > +
> > +    /*
> > +     * NOTE: This will really not work well if those are remapped
> > +     * after the PHB has created its sub regions. We could do better
> > +     * if we had a way to resize regions but we don't really care
> > +     * that much in practice as the stuff below really only happens
> > +     * once early during boot
> > +     */
> > +
> > +    /* Handle unmaps */
> > +    if (pbcq->mmio0_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
> > +        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar0);
> > +        pbcq->mmio0_mapped = false;
> > +    }
> > +    if (pbcq->mmio1_mapped && !(bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
> > +        memory_region_del_subregion(get_system_memory(), &pbcq->mmbar1);
> > +        pbcq->mmio1_mapped = false;
> > +    }
> > +    if (pbcq->phb_mapped && !(bar_en & PBCQ_NEST_BAR_EN_PHB)) {
> > +        memory_region_del_subregion(get_system_memory(), &pbcq->phbbar);
> > +        pbcq->phb_mapped = false;
> > +    }
> > +
> > +    /* Update PHB */
> > +    pnv_phb3_update_regions(pbcq->phb);
> > +
> > +    /* Handle maps */
> > +    if (!pbcq->mmio0_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO0)) {
> > +        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] >> 14;
> > +        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0];
> > +        size = ((~mask) >> 14) + 1;
> > +        memory_region_init(&pbcq->mmbar0, OBJECT(pbcq), "pbcq-mmio0", size);
> > +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar0);
> > +        pbcq->mmio0_mapped = true;
> > +        pbcq->mmio0_base = bar;
> > +        pbcq->mmio0_size = size;
> > +    }
> > +    if (!pbcq->mmio1_mapped && (bar_en & PBCQ_NEST_BAR_EN_MMIO1)) {
> > +        bar = pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] >> 14;
> > +        mask = pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1];
> > +        size = ((~mask) >> 14) + 1;
> > +        memory_region_init(&pbcq->mmbar1, OBJECT(pbcq), "pbcq-mmio1", size);
> > +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->mmbar1);
> > +        pbcq->mmio1_mapped = true;
> > +        pbcq->mmio1_base = bar;
> > +        pbcq->mmio1_size = size;
> > +    }
> > +    if (!pbcq->phb_mapped && (bar_en & PBCQ_NEST_BAR_EN_PHB)) {
> > +        bar = pbcq->nest_regs[PBCQ_NEST_PHB_BAR] >> 14;
> > +        size = 0x1000;
> > +        memory_region_init(&pbcq->phbbar, OBJECT(pbcq), "pbcq-phb", size);
> > +        memory_region_add_subregion(get_system_memory(), bar, &pbcq->phbbar);
> > +        pbcq->phb_mapped = true;
> > +    }
> > +
> > +    /* Update PHB */
> > +    pnv_phb3_update_regions(pbcq->phb);
> > +}
> > +
> > +static void pnv_pbcq_nest_xscom_write(void *opaque, hwaddr addr,
> > +                                uint64_t val, unsigned size)
> > +{
> > +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> > +    uint32_t reg = addr >> 3;
> > +
> > +    switch (reg) {
> > +    case PBCQ_NEST_MMIO_BAR0:
> > +    case PBCQ_NEST_MMIO_BAR1:
> > +    case PBCQ_NEST_MMIO_MASK0:
> > +    case PBCQ_NEST_MMIO_MASK1:
> > +        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] &
> > +            (PBCQ_NEST_BAR_EN_MMIO0 |
> > +             PBCQ_NEST_BAR_EN_MMIO1)) {
> > +            qemu_log_mask(LOG_GUEST_ERROR,
> > +                           "PHB3: Changing enabled BAR unsupported\n");
> > +        }
> > +        pbcq->nest_regs[reg] = val & 0xffffffffc0000000ull;
> > +        break;
> > +    case PBCQ_NEST_PHB_BAR:
> > +        if (pbcq->nest_regs[PBCQ_NEST_BAR_EN] & PBCQ_NEST_BAR_EN_PHB) {
> > +            qemu_log_mask(LOG_GUEST_ERROR,
> > +                           "PHB3: Changing enabled BAR unsupported\n");
> > +        }
> > +        pbcq->nest_regs[reg] = val & 0xfffffffffc000000ull;
> > +        break;
> > +    case PBCQ_NEST_BAR_EN:
> > +        pbcq->nest_regs[reg] = val & 0xf800000000000000ull;
> > +        pnv_pbcq_update_map(pbcq);
> > +        pnv_phb3_remap_irqs(pbcq->phb);
> > +        break;
> > +    case PBCQ_NEST_IRSN_COMPARE:
> > +    case PBCQ_NEST_IRSN_MASK:
> > +        pbcq->nest_regs[reg] = val & PBCQ_NEST_IRSN_COMP;
> > +        pnv_phb3_remap_irqs(pbcq->phb);
> > +        break;
> > +    case PBCQ_NEST_LSI_SRC_ID:
> > +        pbcq->nest_regs[reg] = val & PBCQ_NEST_LSI_SRC;
> > +        pnv_phb3_remap_irqs(pbcq->phb);
> > +        break;
> > +    }
> > +
> > +    /* XXX Don't error out on other regs for now ... */
> > +}
> > +
> > +static void pnv_pbcq_pci_xscom_write(void *opaque, hwaddr addr,
> > +                                     uint64_t val, unsigned size)
> > +{
> > +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> > +    uint32_t reg = addr >> 3;
> > +
> > +    switch (reg) {
> > +    case PBCQ_PCI_BAR2:
> > +        pbcq->pci_regs[reg] = val & 0xfffffffffc000000ull;
> > +        pnv_pbcq_update_map(pbcq);
> > +        break;
> > +    }
> > +
> > +    /* XXX Don't error out on other regs for now ... */
> > +}
> > +
> > +static void pnv_pbcq_spci_xscom_write(void *opaque, hwaddr addr,
> > +                                uint64_t val, unsigned size)
> > +{
> > +    PnvPBCQState *pbcq = PNV_PBCQ(opaque);
> > +    uint32_t reg = addr >> 3;
> > +
> > +    switch (reg) {
> > +    case PBCQ_SPCI_ASB_ADDR:
> > +        pbcq->spci_regs[reg] = val & 0xfff;
> > +        break;
> > +    case PBCQ_SPCI_ASB_STATUS:
> > +        pbcq->spci_regs[reg] &= ~val;
> > +        break;
> > +    case PBCQ_SPCI_ASB_DATA:
> > +        pnv_phb3_reg_write(pbcq->phb, pbcq->spci_regs[PBCQ_SPCI_ASB_ADDR],
> > +                           val, 8);
> > +        break;
> > +    case PBCQ_SPCI_AIB_CAPP_EN:
> > +    case PBCQ_SPCI_CAPP_SEC_TMR:
> > +        break;
> > +    }
> > +
> > +    /* XXX Don't error out on other regs for now ... */
> > +}
> > +
> > +static const MemoryRegionOps pnv_pbcq_nest_xscom_ops = {
> > +    .read = pnv_pbcq_nest_xscom_read,
> > +    .write = pnv_pbcq_nest_xscom_write,
> > +    .valid.min_access_size = 8,
> > +    .valid.max_access_size = 8,
> > +    .impl.min_access_size = 8,
> > +    .impl.max_access_size = 8,
> > +    .endianness = DEVICE_BIG_ENDIAN,
> > +};
> > +
> > +static const MemoryRegionOps pnv_pbcq_pci_xscom_ops = {
> > +    .read = pnv_pbcq_pci_xscom_read,
> > +    .write = pnv_pbcq_pci_xscom_write,
> > +    .valid.min_access_size = 8,
> > +    .valid.max_access_size = 8,
> > +    .impl.min_access_size = 8,
> > +    .impl.max_access_size = 8,
> > +    .endianness = DEVICE_BIG_ENDIAN,
> > +};
> > +
> > +static const MemoryRegionOps pnv_pbcq_spci_xscom_ops = {
> > +    .read = pnv_pbcq_spci_xscom_read,
> > +    .write = pnv_pbcq_spci_xscom_write,
> > +    .valid.min_access_size = 8,
> > +    .valid.max_access_size = 8,
> > +    .impl.min_access_size = 8,
> > +    .impl.max_access_size = 8,
> > +    .endianness = DEVICE_BIG_ENDIAN,
> > +};
> > +
> > +static void pnv_pbcq_default_bars(PnvPBCQState *pbcq)
> > +{
> > +    uint64_t mm0, mm1, reg;
> > +    PnvPHB3 *phb = pbcq->phb;
> > +
> > +    mm0 = 0x3d00000000000ull + 0x4000000000ull * phb->chip_id +
> > +            0x1000000000ull * phb->phb_id;
> > +    mm1 = 0x3ff8000000000ull + 0x0200000000ull * phb->chip_id +
> > +            0x0080000000ull * phb->phb_id;
> > +    reg = 0x3fffe40000000ull + 0x0000400000ull * phb->chip_id +
> > +            0x0000100000ull * phb->phb_id;
> > +
> > +    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR0] = mm0 << 14;
> > +    pbcq->nest_regs[PBCQ_NEST_MMIO_BAR1] = mm1 << 14;
> > +    pbcq->nest_regs[PBCQ_NEST_PHB_BAR] = reg << 14;
> > +    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK0] = 0x3fff000000000ull << 14;
> > +    pbcq->nest_regs[PBCQ_NEST_MMIO_MASK1] = 0x3ffff80000000ull << 14;
> > +    pbcq->pci_regs[PBCQ_PCI_BAR2] = reg << 14;
> > +}
> > +
> > +static void pnv_pbcq_realize(DeviceState *dev, Error **errp)
> > +{
> > +    PnvPBCQState *pbcq = PNV_PBCQ(dev);
> > +    PnvPHB3 *phb;
> > +    Object *obj;
> > +    Error *local_err = NULL;
> > +    char name[32];
> > +
> > +    obj = object_property_get_link(OBJECT(dev), "phb", &local_err);
> > +    if (!obj) {
> > +        error_propagate(errp, local_err);
> > +        error_prepend(errp, "required link 'phb' not found: ");
> > +        return;
> > +    }
> > +    phb = pbcq->phb = PNV_PHB3(obj);
> > +
> > +    /* XXX Fix OPAL to do that: establish default BAR values */
> > +    pnv_pbcq_default_bars(pbcq);
> > +
> > +    /* Initialize the XSCOM region for the PBCQ registers */
> > +    snprintf(name, sizeof(name), "xscom-pbcq-nest-%d.%d",
> > +             phb->chip_id, phb->phb_id);
> > +    pnv_xscom_region_init(&pbcq->xscom_nest_regs, OBJECT(dev),
> > +                          &pnv_pbcq_nest_xscom_ops, pbcq, name,
> > +                          PNV_XSCOM_PBCQ_NEST_SIZE);
> > +    snprintf(name, sizeof(name), "xscom-pbcq-pci-%d.%d",
> > +             phb->chip_id, phb->phb_id);
> > +    pnv_xscom_region_init(&pbcq->xscom_pci_regs, OBJECT(dev),
> > +                          &pnv_pbcq_pci_xscom_ops, pbcq, name,
> > +                          PNV_XSCOM_PBCQ_PCI_SIZE);
> > +    snprintf(name, sizeof(name), "xscom-pbcq-spci-%d.%d",
> > +             phb->chip_id, phb->phb_id);
> > +    pnv_xscom_region_init(&pbcq->xscom_spci_regs, OBJECT(dev),
> > +                          &pnv_pbcq_spci_xscom_ops, pbcq, name,
> > +                          PNV_XSCOM_PBCQ_SPCI_SIZE);
> > +}
> > +
> > +static int pnv_pbcq_dt_xscom(PnvXScomInterface *dev, void *fdt,
> > +                             int xscom_offset)
> > +{
> > +    const char compat[] = "ibm,power8-pbcq";
> > +    PnvPHB3 *phb = PNV_PBCQ(dev)->phb;
> > +    char *name;
> > +    int offset;
> > +    uint32_t lpc_pcba = PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id;
> > +    uint32_t reg[] = {
> > +        cpu_to_be32(lpc_pcba),
> > +        cpu_to_be32(PNV_XSCOM_PBCQ_NEST_SIZE),
> > +        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id),
> > +        cpu_to_be32(PNV_XSCOM_PBCQ_PCI_SIZE),
> > +        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id),
> > +        cpu_to_be32(PNV_XSCOM_PBCQ_SPCI_SIZE)
> > +    };
> > +
> > +    name = g_strdup_printf("pbcq@%x", lpc_pcba);
> > +    offset = fdt_add_subnode(fdt, xscom_offset, name);
> > +    _FDT(offset);
> > +    g_free(name);
> > +
> > +    _FDT((fdt_setprop(fdt, offset, "reg", reg, sizeof(reg))));
> > +
> > +    _FDT((fdt_setprop_cell(fdt, offset, "ibm,phb-index", phb->phb_id)));
> > +    _FDT((fdt_setprop_cell(fdt, offset, "ibm,chip-id", phb->chip_id)));
> > +    _FDT((fdt_setprop(fdt, offset, "compatible", compat,
> > +                      sizeof(compat))));
> > +    return 0;
> > +}
> > +
> > +static void pnv_pbcq_class_init(ObjectClass *klass, void *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    PnvXScomInterfaceClass *xdc = PNV_XSCOM_INTERFACE_CLASS(klass);
> > +
> > +    xdc->dt_xscom = pnv_pbcq_dt_xscom;
> > +
> > +    dc->realize = pnv_pbcq_realize;
> > +}
> > +
> > +static const TypeInfo pnv_pbcq_type_info = {
> > +    .name          = TYPE_PNV_PBCQ,
> > +    .parent        = TYPE_DEVICE,
> > +    .instance_size = sizeof(PnvPBCQState),
> > +    .class_init    = pnv_pbcq_class_init,
> > +    .interfaces    = (InterfaceInfo[]) {
> > +        { TYPE_PNV_XSCOM_INTERFACE },
> > +        { }
> > +    }
> > +};
> > +
> > +static void pnv_pbcq_register_types(void)
> > +{
> > +    type_register_static(&pnv_pbcq_type_info);
> > +}
> > +
> > +type_init(pnv_pbcq_register_types)
> > diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> > index 7401ffe5b01c..77e92d5e2e66 100644
> > --- a/hw/ppc/pnv.c
> > +++ b/hw/ppc/pnv.c
> > @@ -44,6 +44,7 @@
> >  #include "hw/isa/isa.h"
> >  #include "hw/char/serial.h"
> >  #include "hw/timer/mc146818rtc.h"
> > +#include "hw/pci/msi.h"
> >  
> >  #include <libfdt.h>
> >  
> > @@ -555,6 +556,9 @@ static void pnv_init(MachineState *machine)
> >      int i;
> >      char *chip_typename;
> >  
> > +    /* MSIs are supported on this platform */
> > +    msi_nonbroken = true;
> > +
> >      /* allocate RAM */
> >      if (machine->ram_size < (1 * G_BYTE)) {
> >          warn_report("skiboot may not work with < 1GB of RAM");
> > @@ -725,6 +729,8 @@ static Object *pnv_chip_power9_intc_create(PnvChip *chip, Object *child,
> >  static void pnv_chip_power8_instance_init(Object *obj)
> >  {
> >      Pnv8Chip *chip8 = PNV8_CHIP(obj);
> > +    PnvChipClass *pcc = PNV_CHIP_GET_CLASS(obj);
> > +    int i;
> >  
> >      object_initialize(&chip8->psi, sizeof(chip8->psi), TYPE_PNV_PSI);
> >      object_property_add_child(obj, "psi", OBJECT(&chip8->psi), NULL);
> > @@ -740,6 +746,12 @@ static void pnv_chip_power8_instance_init(Object *obj)
> >      object_property_add_child(obj, "occ", OBJECT(&chip8->occ), NULL);
> >      object_property_add_const_link(OBJECT(&chip8->occ), "psi",
> >                                     OBJECT(&chip8->psi), &error_abort);
> > +
> > +    for (i = 0; i < pcc->num_phbs; i++) {
> > +        object_initialize(&chip8->phbs[i], sizeof(chip8->phbs[i]),
> > +                          TYPE_PNV_PHB3);
> > +        object_property_add_child(obj, "phb[*]", OBJECT(&chip8->phbs[i]), NULL);
> > +    }
> >  }
> >  
> >  static void pnv_chip_icp_realize(Pnv8Chip *chip8, Error **errp)
> > @@ -780,6 +792,7 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
> >      PnvChip *chip = PNV_CHIP(dev);
> >      Pnv8Chip *chip8 = PNV8_CHIP(dev);
> >      Error *local_err = NULL;
> > +    int i;
> >  
> >      pcc->parent_realize(dev, &local_err);
> >      if (local_err) {
> > @@ -817,6 +830,33 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
> >          return;
> >      }
> >      pnv_xscom_add_subregion(chip, PNV_XSCOM_OCC_BASE, &chip8->occ.xscom_regs);
> > +
> > +    /* Create the PHB3 controllers */
> > +    for (i = 0; i < pcc->num_phbs; i++) {
> > +        PnvPHB3 *phb = &chip8->phbs[i];
> > +        PnvPBCQState *pbcq = &phb->pbcq;
> > +
> > +        object_property_set_int(OBJECT(phb), i, "phb-id", &error_fatal);
> > +        object_property_set_int(OBJECT(phb), chip->chip_id, "chip-id",
> > +                                &error_fatal);
> > +        object_property_set_bool(OBJECT(phb), true, "realized", &local_err);
> > +        if (local_err) {
> > +            error_propagate(errp, local_err);
> > +            return;
> > +        }
> > +        qdev_set_parent_bus(DEVICE(phb), sysbus_get_default());
> > +
> > +        /* Populate the XSCOM address space. */
> > +        pnv_xscom_add_subregion(chip,
> > +                                PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
> > +                                &pbcq->xscom_nest_regs);
> > +        pnv_xscom_add_subregion(chip,
> > +                                PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
> > +                                &pbcq->xscom_pci_regs);
> > +        pnv_xscom_add_subregion(chip,
> > +                                PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
> > +                                &pbcq->xscom_spci_regs);
> > +    }
> >  }
> >  
> >  static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
> > @@ -827,6 +867,7 @@ static void pnv_chip_power8e_class_init(ObjectClass *klass, void *data)
> >      k->chip_type = PNV_CHIP_POWER8E;
> >      k->chip_cfam_id = 0x221ef04980000000ull;  /* P8 Murano DD2.1 */
> >      k->cores_mask = POWER8E_CORE_MASK;
> > +    k->num_phbs = 2;
> >      k->core_pir = pnv_chip_core_pir_p8;
> >      k->intc_create = pnv_chip_power8_intc_create;
> >      k->isa_create = pnv_chip_power8_isa_create;
> > @@ -845,6 +886,7 @@ static void pnv_chip_power8_class_init(ObjectClass *klass, void *data)
> >      k->chip_type = PNV_CHIP_POWER8;
> >      k->chip_cfam_id = 0x220ea04980000000ull; /* P8 Venice DD2.0 */
> >      k->cores_mask = POWER8_CORE_MASK;
> > +    k->num_phbs = 3;
> >      k->core_pir = pnv_chip_core_pir_p8;
> >      k->intc_create = pnv_chip_power8_intc_create;
> >      k->isa_create = pnv_chip_power8_isa_create;
> > @@ -863,6 +905,7 @@ static void pnv_chip_power8nvl_class_init(ObjectClass *klass, void *data)
> >      k->chip_type = PNV_CHIP_POWER8NVL;
> >      k->chip_cfam_id = 0x120d304980000000ull;  /* P8 Naples DD1.0 */
> >      k->cores_mask = POWER8_CORE_MASK;
> > +    k->num_phbs = 4;
> >      k->core_pir = pnv_chip_core_pir_p8;
> >      k->intc_create = pnv_chip_power8_intc_create;
> >      k->isa_create = pnv_chip_power8nvl_isa_create;
> > @@ -897,6 +940,7 @@ static void pnv_chip_power9_class_init(ObjectClass *klass, void *data)
> >      k->chip_type = PNV_CHIP_POWER9;
> >      k->chip_cfam_id = 0x220d104900008000ull; /* P9 Nimbus DD2.0 */
> >      k->cores_mask = POWER9_CORE_MASK;
> > +    k->num_phbs = 0;
> >      k->core_pir = pnv_chip_core_pir_p9;
> >      k->intc_create = pnv_chip_power9_intc_create;
> >      k->isa_create = pnv_chip_power9_isa_create;
> > @@ -1047,14 +1091,24 @@ static void pnv_chip_class_init(ObjectClass *klass, void *data)
> >  static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
> >  {
> >      PnvMachineState *pnv = PNV_MACHINE(xi);
> > -    int i;
> > +    int i, j;
> >  
> >      for (i = 0; i < pnv->num_chips; i++) {
> > +        PnvChip *chip = pnv->chips[i];
> >          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> > +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
> >  
> >          if (ics_valid_irq(&chip8->psi.ics, irq)) {
> >              return &chip8->psi.ics;
> >          }
> > +        for (j = 0; j < pcc->num_phbs; j++) {
> > +            if (ics_valid_irq(&chip8->phbs[j].lsis, irq)) {
> > +                return &chip8->phbs[j].lsis;
> > +            }
> > +            if (ics_valid_irq(ICS_BASE(&chip8->phbs[j].msis), irq)) {
> > +                return ICS_BASE(&chip8->phbs[j].msis);
> > +            }
> > +        }
> >      }
> >      return NULL;
> >  }
> > @@ -1062,11 +1116,18 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
> >  static void pnv_ics_resend(XICSFabric *xi)
> >  {
> >      PnvMachineState *pnv = PNV_MACHINE(xi);
> > -    int i;
> > +    int i, j;
> >  
> >      for (i = 0; i < pnv->num_chips; i++) {
> > +        PnvChip *chip = pnv->chips[i];
> >          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> > +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
> > +
> >          ics_resend(&chip8->psi.ics);
> > +        for (j = 0; j < pcc->num_phbs; j++) {
> > +            ics_resend(&chip8->phbs[j].lsis);
> > +            ics_resend(ICS_BASE(&chip8->phbs[j].msis));
> > +        }
> >      }
> >  }
> >  
> > @@ -1097,7 +1158,7 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
> >                                 Monitor *mon)
> >  {
> >      PnvMachineState *pnv = PNV_MACHINE(obj);
> > -    int i;
> > +    int i, j;
> >      CPUState *cs;
> >  
> >      CPU_FOREACH(cs) {
> > @@ -1107,8 +1168,16 @@ static void pnv_pic_print_info(InterruptStatsProvider *obj,
> >      }
> >  
> >      for (i = 0; i < pnv->num_chips; i++) {
> > +        PnvChip *chip = pnv->chips[i];
> >          Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
> > +        PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
> > +
> >          ics_pic_print_info(&chip8->psi.ics, mon);
> > +
> > +        for (j = 0; j < pcc->num_phbs; j++) {
> > +            ics_pic_print_info(&chip8->phbs[j].lsis, mon);
> > +            ics_pic_print_info(ICS_BASE(&chip8->phbs[j].msis), mon);
> > +        }
> >      }
> >  }
> >  
> > diff --git a/hw/ppc/pnv_xscom.c b/hw/ppc/pnv_xscom.c
> > index 46fae41f32b0..83f05d06053f 100644
> > --- a/hw/ppc/pnv_xscom.c
> > +++ b/hw/ppc/pnv_xscom.c
> > @@ -252,7 +252,11 @@ int pnv_dt_xscom(PnvChip *chip, void *fdt, int root_offset)
> >      args.fdt = fdt;
> >      args.xscom_offset = xscom_offset;
> >  
> > -    object_child_foreach(OBJECT(chip), xscom_dt_child, &args);
> > +    /* Some PnvXScomInterface objects lie a bit deeper (PnvPBCQState)
> > +     * than the first layer, so we need to loop on the whole object
> > +     * hierarchy to catch them
> > +     */
> > +    object_child_foreach_recursive(OBJECT(chip), xscom_dt_child, &args);
> >      return 0;
> >  }
> >  
> > diff --git a/hw/pci-host/Makefile.objs b/hw/pci-host/Makefile.objs
> > index 6d6597c06563..1b5b782bd986 100644
> > --- a/hw/pci-host/Makefile.objs
> > +++ b/hw/pci-host/Makefile.objs
> > @@ -19,3 +19,4 @@ common-obj-$(CONFIG_PCI_GENERIC) += gpex.o
> >  common-obj-$(CONFIG_PCI_XILINX) += xilinx-pcie.o
> >  
> >  common-obj-$(CONFIG_PCI_DESIGNWARE) += designware.o
> > +obj-$(CONFIG_POWERNV) += pnv_phb3.o pnv_phb3_pbcq.o pnv_phb3_msi.o
> > 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-18  6:12   ` David Gibson
@ 2018-07-18  8:03     ` Benjamin Herrenschmidt
  2018-07-23  4:16       ` David Gibson
  2018-07-23 21:35     ` Cédric Le Goater
                       ` (3 subsequent siblings)
  4 siblings, 1 reply; 29+ messages in thread
From: Benjamin Herrenschmidt @ 2018-07-18  8:03 UTC (permalink / raw)
  To: David Gibson, Cédric Le Goater
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On Wed, 2018-07-18 at 16:12 +1000, David Gibson wrote:
> On Thu, Jun 28, 2018 at 10:36:32AM +0200, Cédric Le Goater wrote:
> > From: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Can you trim your replies ? It's really hard to find your comments in
such a long patch...

> > diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> > index 6ac8a9392da6..966a996c2eac 100644
> > --- a/include/hw/ppc/xics.h
> > +++ b/include/hw/ppc/xics.h
> > @@ -194,6 +194,7 @@ void icp_set_mfrr(ICPState *icp, uint8_t mfrr);
> >  uint32_t icp_accept(ICPState *ss);
> >  uint32_t icp_ipoll(ICPState *ss, uint32_t *mfrr);
> >  void icp_eoi(ICPState *icp, uint32_t xirr);
> > +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority);
> 
> Hrm... are you sure you need to expose this?

See further down...

> > +/*
> > + * The CONFIG_DATA register expects little endian accesses, but as the
> > + * region is big endian, we have to swap the value.
> > + */
> > +static void pnv_phb3_config_write(PnvPHB3 *phb, unsigned off,
> > +                                  unsigned size, uint64_t val)
> > +{
> > +    uint32_t cfg_addr, limit;
> > +    PCIDevice *pdev;
> > +
> > +    pdev = pnv_phb3_find_cfg_dev(phb);
> > +    if (!pdev) {
> > +        return;
> > +    }
> > +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xfff;
> > +    cfg_addr |= off;
> 
> This looks weird - there are callers of this that appear to have low
> bits in 'off', then you're ORing it with overlapping low bits.

Should be ffc like the read case.

> 
> > +    limit = pci_config_size(pdev);
> > +    if (limit <= cfg_addr) {
> > +        /* conventional pci device can be behind pcie-to-pci bridge.
> > +           256 <= addr < 4K has no effects. */
> > +        return;
> > +    }
> > +    switch (size) {
> > +    case 1:
> > +        break;
> > +    case 2:
> > +        val = bswap16(val);
> > +        break;
> > +    case 4:
> > +        val = bswap32(val);
> > +        break;
> > +    default:
> > +        g_assert_not_reached();
> > +    }
> > +    pci_host_config_write_common(pdev, cfg_addr, limit, val, size);
> > +}
> > +
> > +static uint64_t pnv_phb3_config_read(PnvPHB3 *phb, unsigned off,
> > +                                     unsigned size)
> > +{
> > +    uint32_t cfg_addr, limit;
> > +    PCIDevice *pdev;
> > +    uint64_t val;
> > +
> > +    pdev = pnv_phb3_find_cfg_dev(phb);
> > +    if (!pdev) {
> > +        return ~0ull;
> > +    }
> > +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xffc;
> > +    cfg_addr |= off;
> 
> This looks better, should it be 0xffc in the write path as well?
> 
> > +    limit = pci_config_size(pdev);
> > +    if (limit <= cfg_addr) {
> > +        /* conventional pci device can be behind pcie-to-pci bridge.
> > +           256 <= addr < 4K has no effects. */
> > +        return ~0ull;
> > +    }
> > +    val = pci_host_config_read_common(pdev, cfg_addr, limit, size);
> > +    switch (size) {
> > +    case 1:
> > +        return val;
> > +    case 2:
> > +        return bswap16(val);
> > +    case 4:
> > +        return bswap32(val);
> > +    default:
> > +        g_assert_not_reached();
> > +    }
> > +}
> > +
> > +static void pnv_phb3_check_m32(PnvPHB3 *phb)
> > +{
> > +    uint64_t base, start, size;
> > +    MemoryRegion *parent;
> > +    PnvPBCQState *pbcq = &phb->pbcq;
> > +
> > +    if (phb->m32_mapped) {
> > +        memory_region_del_subregion(phb->mr_m32.container, &phb->mr_m32);
> > +        phb->m32_mapped = false;
> 
> Could you use memory_region_set_enabled() rather than having to add
> and delete the subregion and keep track of it in this separate
> variable?

There was a reason for that but it's years old and I forgot... I think
when re-enabled it moves around, among other things. So it's not more
efficient.

> > +    }
> > +
> > +    /* Disabled ? move on with life ... */
> 
> Trivial: "nothing to do" seems to be the standard way to express this.
> Even trivialler: usual English rules don't put a space before '?' or
> '!' punctuation.

No, that's the tasteless English rule :-) It shall be overridden by
anybody interested in making things actually readable :-)

> > +
> > +static void pnv_phb3_lxivt_write(PnvPHB3 *phb, unsigned idx, uint64_t val)
> > +{
> > +    ICSState *ics = &phb->lsis;
> > +    uint8_t server, prio;
> > +
> > +    phb->ioda_LXIVT[idx] = val & (IODA2_LXIVT_SERVER |
> > +                                  IODA2_LXIVT_PRIORITY |
> > +                                  IODA2_LXIVT_NODE_ID);
> > +    server = GETFIELD(IODA2_LXIVT_SERVER, val);
> > +    prio = GETFIELD(IODA2_LXIVT_PRIORITY, val);
> > +
> > +    /*
> > +     * The low order 2 bits are the link pointer (Type II interrupts).
> 
> I don't think we've currently implemented link pointers in our xics
> code.  Do we need to do that?

Not until somebody uses them (other than pHyp).

> > +     * Shift back to get a valid IRQ server.
> > +     */
> > +    server >>= 2;
> > +
> > +    ics_simple_write_xive(ics, idx, server, prio, prio);
> > +}
> > +
> > +static uint64_t *pnv_phb3_ioda_access(PnvPHB3 *phb,
> > +                                      unsigned *out_table, unsigned *out_idx)
> > +{
> > +    uint64_t adreg = phb->regs[PHB_IODA_ADDR >> 3];
> > +    unsigned int index = GETFIELD(PHB_IODA_AD_TADR, adreg);
> > +    unsigned int table = GETFIELD(PHB_IODA_AD_TSEL, adreg);
> > +    unsigned int mask;
> > +    uint64_t *tptr = NULL;
> > +
> > +    switch (table) {
> > +    case IODA2_TBL_LIST:
> > +        tptr = phb->ioda_LIST;
> > +        mask = 7;
> 
> I'd suggest hex for the masks.

This is more readable when matched with the documentation but not a big
deal.

> > +        break;
> > +    case IODA2_TBL_LXIVT:
> > +        tptr = phb->ioda_LXIVT;
> > +        mask = 7;
> > +        break;
> > +    case IODA2_TBL_IVC_CAM:
> > +    case IODA2_TBL_RBA:
> > +        mask = 31;
> > +        break;
> > +    case IODA2_TBL_RCAM:
> > +        mask = 63;
> > +        break;
> > +    case IODA2_TBL_MRT:
> > +        mask = 7;
> > +        break;
> > +    case IODA2_TBL_PESTA:
> > +    case IODA2_TBL_PESTB:
> > +        mask = 255;
> > +        break;
> > +    case IODA2_TBL_TVT:
> > +        tptr = phb->ioda_TVT;
> > +        mask = 511;
> > +        break;
> > +    case IODA2_TBL_TCAM:
> > +    case IODA2_TBL_TDR:
> > +        mask = 63;
> > +        break;
> > +    case IODA2_TBL_M64BT:
> > +        tptr = phb->ioda_M64BT;
> > +        mask = 15;
> > +        break;
> > +    case IODA2_TBL_M32DT:
> > +        tptr = phb->ioda_MDT;
> > +        mask = 255;
> > +        break;
> > +    case IODA2_TBL_PEEV:
> > +        tptr = phb->ioda_PEEV;
> > +        mask = 3;
> > +        break;
> > +    default:
> > +        return NULL;
> > +    }
> > +    index &= mask;
> 
> Do we want to silently mask, rather than logging an error if there's
> an access out of bounds for a particular table?

We could do both, as to behave like the HW but also flag an error. But
you have to be careful with the auto-increment below.

> > +    if (out_idx) {
> > +        *out_idx = index;
> > +    }
> > +    if (out_table) {
> > +        *out_table = table;
> > +    }
> > +    if (adreg & PHB_IODA_AD_AUTOINC) {
> > +        index = (index + 1) & mask;
> > +        adreg = SETFIELD(PHB_IODA_AD_TADR, adreg, index);
> > +    }
> > +    if (tptr) {
> > +        tptr += index;
> > +    }
> > +    phb->regs[PHB_IODA_ADDR >> 3] = adreg;
> > +    return tptr;
> > +}

 ../..

> > +    if ((comp & mask) != comp) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "IRQ compare bits not in mask: comp=0x%x mask=0x%x",
> > +                      comp, mask);
> > +        comp &= mask;
> > +    }
> > +    /* Setup LSI offset */
> > +    ics->offset = comp + global;
> 
> Oh.. changing ICS offset at runtime.  I hadn't considered that case..

As above, see further down.

> > +    /* Special case configuration data */
> > +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> > +        pnv_phb3_config_write(phb, off & 0x3, size, val);
> > +        return;
> > +    }
> > +
> > +    /* Other registers are 64-bit only */
> > +    if (size != 8 || off & 0x7) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> > +                      off, size);
> > +        return;
> > +    }
> > +
> > +    /* Handle masking */
> > +    switch (off) {
> > +    case PHB_M64_UPPER_BITS:
> 
> Couldn't you handle this in the switch below - it should be ok to
> momentarily have the invalid bits set in your reg case, as long as you
> mask them before applying the side-effects, yes?

That felt easier that way ...

> That said... shouldn't you filter our invalid or read-only regs before
> updating the cache?

Well, I had a reason for doing it that way, I do have a vague memory of
that but I can't remember it now :-)

> > +/*
> > + * MSI/MSIX memory region implementation.
> > + * The handler handles both MSI and MSIX.
> > + */
> > +static void pnv_phb3_msi_write(void *opaque, hwaddr addr,
> > +                               uint64_t data, unsigned size)
> > +{
> > +    PnvPhb3DMASpace *ds = opaque;
> > +
> > +    /* Resolve PE# */
> > +    if (!pnv_phb3_resolve_pe(ds)) {
> > +        qemu_log_mask(LOG_GUEST_ERROR,
> > +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> > +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> > +        return;
> > +    }
> > +
> > +    pnv_phb3_msi_send(&ds->phb->msis, addr, data, ds->pe_num);
> > +}
> > +
> > +static const MemoryRegionOps pnv_phb3_msi_ops = {
> > +    /* There is no .read as the read result is undefined by PCI spec */
> 
> What will qemu do if it hits a NULL read here?  The behaviour may be
> undefind, but we'd prefer it didn't crash qemu..

Yeah.

> > +    .read = NULL,
> > +    .write = pnv_phb3_msi_write,
> > +    .endianness = DEVICE_LITTLE_ENDIAN
> > +};
> > +
> > +static AddressSpace *pnv_phb3_dma_iommu(PCIBus *bus, void *opaque, int devfn)
> > +{
> > +    PnvPHB3 *phb = opaque;
> > +    PnvPhb3DMASpace *ds;
> > +
> > +    QLIST_FOREACH(ds, &phb->dma_spaces, list) {
> > +        if (ds->bus == bus && ds->devfn == devfn) {
> > +            break;
> > +        }
> > +    }
> > +
> > +    if (ds == NULL) {
> > +        ds = g_malloc0(sizeof(PnvPhb3DMASpace));
> > +        ds->bus = bus;
> > +        ds->devfn = devfn;
> > +        ds->pe_num = PHB_INVALID_PE;
> > +        ds->phb = phb;
> > +        memory_region_init_iommu(&ds->dma_mr, sizeof(ds->dma_mr),
> > +                                 TYPE_PNV_PHB3_IOMMU_MEMORY_REGION,
> > +                                 OBJECT(phb), "phb3_iommu", UINT64_MAX);
> > +        address_space_init(&ds->dma_as, MEMORY_REGION(&ds->dma_mr),
> > +                           "phb3_iommu");
> > +        memory_region_init_io(&ds->msi32_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> > +                              ds, "msi32", 0x10000);
> > +        memory_region_init_io(&ds->msi64_mr, OBJECT(phb), &pnv_phb3_msi_ops,
> > +                              ds, "msi64", 0x100000);
> > +        pnv_phb3_update_msi_regions(ds);
> > +
> > +        QLIST_INSERT_HEAD(&phb->dma_spaces, ds, list);
> > +    }
> > +    return &ds->dma_as;
> > +}
> > +
> > +static void pnv_phb3_instance_init(Object *obj)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(obj);
> > +
> > +    /* Create LSI source */
> > +    object_initialize(&phb->lsis, sizeof(phb->lsis), TYPE_ICS_SIMPLE);
> > +    object_property_add_child(obj, "ics-phb-lsi", OBJECT(&phb->lsis), NULL);
> > +
> > +    /* Default init ... will be fixed by HW inits */
> > +    phb->lsis.offset = 0;
> > +
> > +    /* Create MSI source */
> > +    object_initialize(&phb->msis, sizeof(phb->msis), TYPE_PHB3_MSI);
> > +    object_property_add_const_link(OBJECT(&phb->msis), "phb", obj,
> > +                                   &error_abort);
> > +    object_property_add_child(obj, "ics-phb-msi", OBJECT(&phb->msis), NULL);
> > +
> > +    /* Create PBCQ */
> > +    object_initialize(&phb->pbcq, sizeof(phb->pbcq), TYPE_PNV_PBCQ);
> > +    object_property_add_const_link(OBJECT(&phb->pbcq), "phb", obj,
> > +                                   &error_abort);
> > +    object_property_add_child(obj, "pbcq", OBJECT(&phb->pbcq), NULL);
> > +
> > +    QLIST_INIT(&phb->dma_spaces);
> > +}
> > +
> > +/*
> > + * This could be done under pnv_pbcq_realize
> > + */
> > +static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
> > +{
> > +    PCIHostState *pcih = PCI_HOST_BRIDGE(phb);
> > +    PCIDevice *brdev;
> > +    PCIDevice *pdev;
> > +    PCIBus *parent;
> > +    uint8_t chassis = phb->chip_id * 4 + phb->phb_id;
> > +    uint8_t chassis_nr = 128;
> > +
> > +    /* Add root complex */
> > +    pdev = pci_create(pcih->bus, 0, TYPE_PNV_PHB3_RC);
> > +    object_property_add_child(OBJECT(phb), "phb3-rc", OBJECT(pdev), NULL);
> > +    qdev_prop_set_uint8(DEVICE(pdev), "chassis", chassis);
> > +    qdev_prop_set_uint16(DEVICE(pdev), "slot", 1);
> > +    qdev_init_nofail(DEVICE(pdev));
> > +
> > +    /* Setup bus for that chip */
> > +    parent = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
> > +
> > +    brdev = pci_create(parent, 0, "pci-bridge");
> 
> What is this pci bridge representing?  I know PCI-e PHBs typically
> have a pseudo P2P bridge right under them, but isn't that represnted
> by the root complex above?
> 
> > +    object_property_add_child(OBJECT(parent), "pci-bridge", OBJECT(brdev),
> > +                              NULL);
> > +    qdev_prop_set_uint8(DEVICE(brdev), PCI_BRIDGE_DEV_PROP_CHASSIS_NR,
> > +                        chassis_nr);
> > +    qdev_init_nofail(DEVICE(brdev));
> > +}
> > +
> > +static void pnv_phb3_realize(DeviceState *dev, Error **errp)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(dev);
> > +    PCIHostState *pci = PCI_HOST_BRIDGE(dev);
> > +    Object *xics = OBJECT(qdev_get_machine());
> > +    Error *local_err = NULL;
> > +    int i;
> > +
> > +    memory_region_init(&phb->pci_mmio, OBJECT(phb), "pci-mmio",
> > +                       PCI_MMIO_TOTAL_SIZE);
> > +
> > +    /* PHB3 doesn't support IO space. However, qemu gets very upset if
> > +     * we don't have an IO region to anchor IO BARs onto so we just
> > +     * initialize one which we never hook up to anything
> > +     */
> > +    memory_region_init(&phb->pci_io, OBJECT(phb), "pci-io", 0x10000);
> > +
> > +    memory_region_init_io(&phb->mr_regs, OBJECT(phb), &pnv_phb3_reg_ops, phb,
> > +                          "phb3-regs", 0x1000);
> > +
> > +    object_property_set_int(OBJECT(&phb->lsis), PNV_PHB3_NUM_LSI, "nr-irqs",
> > +                            &error_abort);
> > +    object_property_add_const_link(OBJECT(&phb->lsis), "xics", xics,
> > +                                   &error_abort);
> > +    object_property_set_bool(OBJECT(&phb->lsis), true, "realized", &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    for (i = 0; i < PNV_PHB3_NUM_LSI; i++) {
> > +        ics_set_irq_type(&phb->lsis, i, true);
> > +    }
> > +
> > +    object_property_add_const_link(OBJECT(&phb->msis), "xics", xics,
> > +                                   &error_abort);
> > +    object_property_set_int(OBJECT(&phb->msis), PHB3_MAX_MSI, "nr-irqs",
> > +                            &error_abort);
> > +    object_property_set_bool(OBJECT(&phb->msis), true, "realized", &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    object_property_set_bool(OBJECT(&phb->pbcq), true, "realized", &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    pci->bus = pci_register_root_bus(dev, "phb3-root-bus",
> > +                                pnv_phb3_set_irq, pnv_phb3_map_irq, phb,
> > +                                &phb->pci_mmio, &phb->pci_io,
> > +                                0, 4, TYPE_PNV_PHB3_ROOT_BUS);
> > +    pci_setup_iommu(pci->bus, pnv_phb3_dma_iommu, phb);
> > +
> > +    /* Setup the PCI busses */
> > +    pnv_phb3_pci_create(phb, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +}
> > +
> > +void pnv_phb3_update_regions(PnvPHB3 *phb)
> > +{
> > +    PnvPBCQState *pbcq = &phb->pbcq;
> > +
> > +    /* Unmap first always */
> > +    if (phb->regs_mapped) {
> > +        memory_region_del_subregion(&pbcq->phbbar, &phb->mr_regs);
> > +        phb->regs_mapped = false;
> > +    }
> > +
> > +    /* Map registers if enabled */
> > +    if (pbcq->phb_mapped) {
> > +        /* XXX We should use the PHB BAR 2 register but we don't ... */
> > +        memory_region_add_subregion(&pbcq->phbbar, 0, &phb->mr_regs);
> > +        phb->regs_mapped = true;
> > +    }
> > +
> > +    /* Check/update m32 */
> > +    if (phb->m32_mapped) {
> > +        pnv_phb3_check_m32(phb);
> > +    }
> > +}
> > +
> > +static const char *pnv_phb3_root_bus_path(PCIHostState *host_bridge,
> > +                                          PCIBus *rootbus)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(host_bridge);
> > +
> > +    snprintf(phb->bus_path, sizeof(phb->bus_path), "00%02x:%02x",
> > +             phb->chip_id, phb->phb_id);
> > +    return phb->bus_path;
> > +}
> > +
> > +static void pnv_phb3_get_phb_id(Object *obj, Visitor *v, const char *name,
> > +                              void *opaque, Error **errp)
> > +{
> > +    Property *prop = opaque;
> > +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> > +
> > +    visit_type_uint32(v, name, ptr, errp);
> > +}
> > +
> > +static void pnv_phb3_set_phb_id(Object *obj, Visitor *v, const char *name,
> > +                              void *opaque, Error **errp)
> > +{
> > +    PnvPHB3 *phb = PNV_PHB3(obj);
> > +    uint32_t phb_id;
> > +    Error *local_err = NULL;
> > +
> > +    visit_type_uint32(v, name, &phb_id, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    /*
> > +     * Limit to a maximum of 6 PHBs per chip
> > +     */
> > +    if (phb_id >= PNV8_CHIP_PHB3_MAX) {
> > +        error_setg(errp, "invalid PHB index: '%d'", phb_id);
> > +        return;
> > +    }
> > +
> > +    phb->phb_id = phb_id;
> > +}
> > +
> > +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
> > +    .name = "irq",
> > +    .get = pnv_phb3_get_phb_id,
> > +    .set = pnv_phb3_set_phb_id,
> > +};
> 
> Can't you use a static DeviceProps style property for this, which is a
> bit simpler?
> 
> > +    /*
> > +     * The low order 2 bits are the link pointer (Type II interrupts).
> > +     * Shift back to get a valid IRQ server.
> > +     */
> > +    server >>= 2;
> > +
> > +    switch (pq) {
> > +    case 0: /* 00 */
> > +        if (prio == 0xff) {
> > +            /* Masked, set Q */
> > +            phb3_msi_set_q(msi, srcno);
> > +        } else {
> > +            /* Enabled, set P and send */
> > +            phb3_msi_set_p(msi, srcno, gen);
> > +            icp_irq(ics, server, srcno + ics->offset, prio);
> 
> Can't you just pulse the right qirq here, which will use the core ICS
> logic to propagate to the ICP?

Ok, interrupts ... sooooo...

This code predates a pile of XICS work you guys did to start with.

Now, this is an ICS subclass, so why shouldn't it directly poke at the
target ICP ? It's an alternate to the normal ICS since it behaves a bit
differently (PQ bits & all).

Cheers,
Ben.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-18  8:03     ` Benjamin Herrenschmidt
@ 2018-07-23  4:16       ` David Gibson
  2018-07-23 23:55         ` Benjamin Herrenschmidt
  0 siblings, 1 reply; 29+ messages in thread
From: David Gibson @ 2018-07-23  4:16 UTC (permalink / raw)
  To: Benjamin Herrenschmidt
  Cc: Cédric Le Goater, qemu-ppc, qemu-devel, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

[-- Attachment #1: Type: text/plain, Size: 10461 bytes --]

On Wed, Jul 18, 2018 at 06:03:29PM +1000, Benjamin Herrenschmidt wrote:
> On Wed, 2018-07-18 at 16:12 +1000, David Gibson wrote:
> > On Thu, Jun 28, 2018 at 10:36:32AM +0200, Cédric Le Goater wrote:
> > > From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> 
> Can you trim your replies ? It's really hard to find your comments in
> such a long patch...
> 
> > > diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> > > index 6ac8a9392da6..966a996c2eac 100644
> > > --- a/include/hw/ppc/xics.h
> > > +++ b/include/hw/ppc/xics.h
> > > @@ -194,6 +194,7 @@ void icp_set_mfrr(ICPState *icp, uint8_t mfrr);
> > >  uint32_t icp_accept(ICPState *ss);
> > >  uint32_t icp_ipoll(ICPState *ss, uint32_t *mfrr);
> > >  void icp_eoi(ICPState *icp, uint32_t xirr);
> > > +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority);
> > 
> > Hrm... are you sure you need to expose this?

[snip]
> > > +    limit = pci_config_size(pdev);
> > > +    if (limit <= cfg_addr) {
> > > +        /* conventional pci device can be behind pcie-to-pci bridge.
> > > +           256 <= addr < 4K has no effects. */
> > > +        return ~0ull;
> > > +    }
> > > +    val = pci_host_config_read_common(pdev, cfg_addr, limit, size);
> > > +    switch (size) {
> > > +    case 1:
> > > +        return val;
> > > +    case 2:
> > > +        return bswap16(val);
> > > +    case 4:
> > > +        return bswap32(val);
> > > +    default:
> > > +        g_assert_not_reached();
> > > +    }
> > > +}
> > > +
> > > +static void pnv_phb3_check_m32(PnvPHB3 *phb)
> > > +{
> > > +    uint64_t base, start, size;
> > > +    MemoryRegion *parent;
> > > +    PnvPBCQState *pbcq = &phb->pbcq;
> > > +
> > > +    if (phb->m32_mapped) {
> > > +        memory_region_del_subregion(phb->mr_m32.container, &phb->mr_m32);
> > > +        phb->m32_mapped = false;
> > 
> > Could you use memory_region_set_enabled() rather than having to add
> > and delete the subregion and keep track of it in this separate
> > variable?
> 
> There was a reason for that but it's years old and I forgot... I think
> when re-enabled it moves around, among other things. So it's not more
> efficient.

Hm, ok.  It'd be good to know what this is.

> > > +    }
> > > +
> > > +    /* Disabled ? move on with life ... */
> > 
> > Trivial: "nothing to do" seems to be the standard way to express this.
> > Even trivialler: usual English rules don't put a space before '?' or
> > '!' punctuation.
> 
> No, that's the tasteless English rule :-) It shall be overridden by
> anybody interested in making things actually readable :-)
> 
> > > +
> > > +static void pnv_phb3_lxivt_write(PnvPHB3 *phb, unsigned idx, uint64_t val)
> > > +{
> > > +    ICSState *ics = &phb->lsis;
> > > +    uint8_t server, prio;
> > > +
> > > +    phb->ioda_LXIVT[idx] = val & (IODA2_LXIVT_SERVER |
> > > +                                  IODA2_LXIVT_PRIORITY |
> > > +                                  IODA2_LXIVT_NODE_ID);
> > > +    server = GETFIELD(IODA2_LXIVT_SERVER, val);
> > > +    prio = GETFIELD(IODA2_LXIVT_PRIORITY, val);
> > > +
> > > +    /*
> > > +     * The low order 2 bits are the link pointer (Type II interrupts).
> > 
> > I don't think we've currently implemented link pointers in our xics
> > code.  Do we need to do that?
> 
> Not until somebody uses them (other than pHyp).
> 
> > > +     * Shift back to get a valid IRQ server.
> > > +     */
> > > +    server >>= 2;
> > > +
> > > +    ics_simple_write_xive(ics, idx, server, prio, prio);
> > > +}
> > > +
> > > +static uint64_t *pnv_phb3_ioda_access(PnvPHB3 *phb,
> > > +                                      unsigned *out_table, unsigned *out_idx)
> > > +{
> > > +    uint64_t adreg = phb->regs[PHB_IODA_ADDR >> 3];
> > > +    unsigned int index = GETFIELD(PHB_IODA_AD_TADR, adreg);
> > > +    unsigned int table = GETFIELD(PHB_IODA_AD_TSEL, adreg);
> > > +    unsigned int mask;
> > > +    uint64_t *tptr = NULL;
> > > +
> > > +    switch (table) {
> > > +    case IODA2_TBL_LIST:
> > > +        tptr = phb->ioda_LIST;
> > > +        mask = 7;
> > 
> > I'd suggest hex for the masks.
> 
> This is more readable when matched with the documentation but not a big
> deal.

Matching the docs is a good enough reason to keep decimal.

> > > +        break;
> > > +    case IODA2_TBL_LXIVT:
> > > +        tptr = phb->ioda_LXIVT;
> > > +        mask = 7;
> > > +        break;
> > > +    case IODA2_TBL_IVC_CAM:
> > > +    case IODA2_TBL_RBA:
> > > +        mask = 31;
> > > +        break;
> > > +    case IODA2_TBL_RCAM:
> > > +        mask = 63;
> > > +        break;
> > > +    case IODA2_TBL_MRT:
> > > +        mask = 7;
> > > +        break;
> > > +    case IODA2_TBL_PESTA:
> > > +    case IODA2_TBL_PESTB:
> > > +        mask = 255;
> > > +        break;
> > > +    case IODA2_TBL_TVT:
> > > +        tptr = phb->ioda_TVT;
> > > +        mask = 511;
> > > +        break;
> > > +    case IODA2_TBL_TCAM:
> > > +    case IODA2_TBL_TDR:
> > > +        mask = 63;
> > > +        break;
> > > +    case IODA2_TBL_M64BT:
> > > +        tptr = phb->ioda_M64BT;
> > > +        mask = 15;
> > > +        break;
> > > +    case IODA2_TBL_M32DT:
> > > +        tptr = phb->ioda_MDT;
> > > +        mask = 255;
> > > +        break;
> > > +    case IODA2_TBL_PEEV:
> > > +        tptr = phb->ioda_PEEV;
> > > +        mask = 3;
> > > +        break;
> > > +    default:
> > > +        return NULL;
> > > +    }
> > > +    index &= mask;
> > 
> > Do we want to silently mask, rather than logging an error if there's
> > an access out of bounds for a particular table?
> 
> We could do both, as to behave like the HW but also flag an error.

Sounds reasonable.

> But
> you have to be careful with the auto-increment below.

Hm, ok.

> > > +    if (out_idx) {
> > > +        *out_idx = index;
> > > +    }
> > > +    if (out_table) {
> > > +        *out_table = table;
> > > +    }
> > > +    if (adreg & PHB_IODA_AD_AUTOINC) {
> > > +        index = (index + 1) & mask;
> > > +        adreg = SETFIELD(PHB_IODA_AD_TADR, adreg, index);
> > > +    }
> > > +    if (tptr) {
> > > +        tptr += index;
> > > +    }
> > > +    phb->regs[PHB_IODA_ADDR >> 3] = adreg;
> > > +    return tptr;
> > > +}
> 
>  ../..
> 
> > > +    if ((comp & mask) != comp) {
> > > +        qemu_log_mask(LOG_GUEST_ERROR,
> > > +                      "IRQ compare bits not in mask: comp=0x%x mask=0x%x",
> > > +                      comp, mask);
> > > +        comp &= mask;
> > > +    }
> > > +    /* Setup LSI offset */
> > > +    ics->offset = comp + global;
> > 
> > Oh.. changing ICS offset at runtime.  I hadn't considered that case..
> 
> As above, see further down.
> 
> > > +    /* Special case configuration data */
> > > +    if ((off & 0xfffc) == PHB_CONFIG_DATA) {
> > > +        pnv_phb3_config_write(phb, off & 0x3, size, val);
> > > +        return;
> > > +    }
> > > +
> > > +    /* Other registers are 64-bit only */
> > > +    if (size != 8 || off & 0x7) {
> > > +        qemu_log_mask(LOG_GUEST_ERROR,
> > > +                      "Invalid register access, offset: 0x%"PRIx64" size: %d",
> > > +                      off, size);
> > > +        return;
> > > +    }
> > > +
> > > +    /* Handle masking */
> > > +    switch (off) {
> > > +    case PHB_M64_UPPER_BITS:
> > 
> > Couldn't you handle this in the switch below - it should be ok to
> > momentarily have the invalid bits set in your reg case, as long as you
> > mask them before applying the side-effects, yes?
> 
> That felt easier that way ...
> 
> > That said... shouldn't you filter our invalid or read-only regs before
> > updating the cache?
> 
> Well, I had a reason for doing it that way, I do have a vague memory of
> that but I can't remember it now :-)
> 
> > > +/*
> > > + * MSI/MSIX memory region implementation.
> > > + * The handler handles both MSI and MSIX.
> > > + */
> > > +static void pnv_phb3_msi_write(void *opaque, hwaddr addr,
> > > +                               uint64_t data, unsigned size)
> > > +{
> > > +    PnvPhb3DMASpace *ds = opaque;
> > > +
> > > +    /* Resolve PE# */
> > > +    if (!pnv_phb3_resolve_pe(ds)) {
> > > +        qemu_log_mask(LOG_GUEST_ERROR,
> > > +                      "Failed to resolve PE# for bus @%p (%d) devfn 0x%x",
> > > +                      ds->bus, pci_bus_num(ds->bus), ds->devfn);
> > > +        return;
> > > +    }
> > > +
> > > +    pnv_phb3_msi_send(&ds->phb->msis, addr, data, ds->pe_num);
> > > +}
> > > +
> > > +static const MemoryRegionOps pnv_phb3_msi_ops = {
> > > +    /* There is no .read as the read result is undefined by PCI spec */
> > 
> > What will qemu do if it hits a NULL read here?  The behaviour may be
> > undefind, but we'd prefer it didn't crash qemu..
> 
> Yeah.

[snip]
> > > +    /*
> > > +     * The low order 2 bits are the link pointer (Type II interrupts).
> > > +     * Shift back to get a valid IRQ server.
> > > +     */
> > > +    server >>= 2;
> > > +
> > > +    switch (pq) {
> > > +    case 0: /* 00 */
> > > +        if (prio == 0xff) {
> > > +            /* Masked, set Q */
> > > +            phb3_msi_set_q(msi, srcno);
> > > +        } else {
> > > +            /* Enabled, set P and send */
> > > +            phb3_msi_set_p(msi, srcno, gen);
> > > +            icp_irq(ics, server, srcno + ics->offset, prio);
> > 
> > Can't you just pulse the right qirq here, which will use the core ICS
> > logic to propagate to the ICP?
> 
> Ok, interrupts ... sooooo...
> 
> This code predates a pile of XICS work you guys did to start with.
> 
> Now, this is an ICS subclass, so why shouldn't it directly poke at the
> target ICP ?

That's ok in theory, but causing it to expose the icp interface to a
new module isn't great.

> It's an alternate to the normal ICS since it behaves a bit
> differently (PQ bits & all).

AFAICT the PQ bits are effectively another filtering layer on top of
the same ICS logic as elsewhere.  I think it's better if we can share
that shared logic, rather than replicating it.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-18  6:12   ` David Gibson
  2018-07-18  8:03     ` Benjamin Herrenschmidt
@ 2018-07-23 21:35     ` Cédric Le Goater
  2018-07-23 21:37     ` Cédric Le Goater
                       ` (2 subsequent siblings)
  4 siblings, 0 replies; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-23 21:35 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

>> diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
>> index 6ac8a9392da6..966a996c2eac 100644
>> --- a/include/hw/ppc/xics.h
>> +++ b/include/hw/ppc/xics.h
>> @@ -194,6 +194,7 @@ void icp_set_mfrr(ICPState *icp, uint8_t mfrr);
>>  uint32_t icp_accept(ICPState *ss);
>>  uint32_t icp_ipoll(ICPState *ss, uint32_t *mfrr);
>>  void icp_eoi(ICPState *icp, uint32_t xirr);
>> +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority);
> 
> Hrm... are you sure you need to expose this?
> 
>>  void ics_simple_write_xive(ICSState *ics, int nr, int server,
>>                             uint8_t priority, uint8_t saved_priority);
>> diff --git a/hw/intc/xics.c b/hw/intc/xics.c
>> index b9f1a3c97214..59e2a5217dcc 100644
>> --- a/hw/intc/xics.c
>> +++ b/hw/intc/xics.c
>> @@ -230,7 +230,7 @@ void icp_eoi(ICPState *icp, uint32_t xirr)
>>      }
>>  }
>>  
>> -static void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
>> +void icp_irq(ICSState *ics, int server, int nr, uint8_t priority)
>>  {
>>      ICPState *icp = xics_icp_get(ics->xics, server);
>>  

[ ... ]

>> +
>> +static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
>> +{
>> +    ICSState *ics = ICS_BASE(msi);
>> +    uint64_t ive;
>> +    uint64_t server, prio, pq, gen;
>> +
>> +    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
>> +        return;
>> +    }
>> +
>> +    server = GETFIELD(IODA2_IVT_SERVER, ive);
>> +    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
>> +    pq = GETFIELD(IODA2_IVT_Q, ive);
>> +    if (!ignore_p) {
>> +        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
>> +    }
>> +    gen = GETFIELD(IODA2_IVT_GEN, ive);
>> +
>> +    /*
>> +     * The low order 2 bits are the link pointer (Type II interrupts).
>> +     * Shift back to get a valid IRQ server.
>> +     */
>> +    server >>= 2;
>> +
>> +    switch (pq) {
>> +    case 0: /* 00 */
>> +        if (prio == 0xff) {
>> +            /* Masked, set Q */
>> +            phb3_msi_set_q(msi, srcno);
>> +        } else {
>> +            /* Enabled, set P and send */
>> +            phb3_msi_set_p(msi, srcno, gen);
>> +            icp_irq(ics, server, srcno + ics->offset, prio);
> 
> Can't you just pulse the right qirq here, which will use the core ICS
> logic to propagate to the ICP?

The interrupt vector entries are maintained in the guest memory. We don't 
have an entry in the device model (hcall, mmio on a register) to update 
an IRQ state. So we need to find the target when the IRQ is triggered.  

The interrupt source unit logic of the phb3 device is closer to the Type 3 
architecture (XIVE) in fact.

C.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-18  6:12   ` David Gibson
  2018-07-18  8:03     ` Benjamin Herrenschmidt
  2018-07-23 21:35     ` Cédric Le Goater
@ 2018-07-23 21:37     ` Cédric Le Goater
  2018-07-24  1:29       ` David Gibson
  2018-07-23 21:38     ` Cédric Le Goater
  2018-07-24 12:18     ` Cédric Le Goater
  4 siblings, 1 reply; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-23 21:37 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

On 07/18/2018 08:12 AM, David Gibson wrote:
>> +static void pnv_phb3_get_phb_id(Object *obj, Visitor *v, const char *name,
>> +                              void *opaque, Error **errp)
>> +{
>> +    Property *prop = opaque;
>> +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
>> +
>> +    visit_type_uint32(v, name, ptr, errp);
>> +}
>> +
>> +static void pnv_phb3_set_phb_id(Object *obj, Visitor *v, const char *name,
>> +                              void *opaque, Error **errp)
>> +{
>> +    PnvPHB3 *phb = PNV_PHB3(obj);
>> +    uint32_t phb_id;
>> +    Error *local_err = NULL;
>> +
>> +    visit_type_uint32(v, name, &phb_id, &local_err);
>> +    if (local_err) {
>> +        error_propagate(errp, local_err);
>> +        return;
>> +    }
>> +
>> +    /*
>> +     * Limit to a maximum of 6 PHBs per chip
>> +     */
>> +    if (phb_id >= PNV8_CHIP_PHB3_MAX) {
>> +        error_setg(errp, "invalid PHB index: '%d'", phb_id);
>> +        return;
>> +    }
>> +
>> +    phb->phb_id = phb_id;
>> +}
>> +
>> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
>> +    .name = "irq",
>> +    .get = pnv_phb3_get_phb_id,
>> +    .set = pnv_phb3_set_phb_id,
>> +};
> Can't you use a static DeviceProps style property for this, which is a
> bit simpler?

yes but we will need these ops for user creatable devices. This is where
we will check the chip id.

C. 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-18  6:12   ` David Gibson
                       ` (2 preceding siblings ...)
  2018-07-23 21:37     ` Cédric Le Goater
@ 2018-07-23 21:38     ` Cédric Le Goater
  2018-07-24 12:18     ` Cédric Le Goater
  4 siblings, 0 replies; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-23 21:38 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

On 07/18/2018 08:12 AM, David Gibson wrote:
>> +    /* Setup LSI offset */
>> +    ics->offset = comp + global;
> Oh.. changing ICS offset at runtime.  I hadn't considered that case..

Yes. The PowerNV FW defines the IRQ layout in the overall number space.

C. 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-23  4:16       ` David Gibson
@ 2018-07-23 23:55         ` Benjamin Herrenschmidt
  2018-07-24  2:14           ` David Gibson
  0 siblings, 1 reply; 29+ messages in thread
From: Benjamin Herrenschmidt @ 2018-07-23 23:55 UTC (permalink / raw)
  To: David Gibson
  Cc: Cédric Le Goater, qemu-ppc, qemu-devel, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

On Mon, 2018-07-23 at 14:16 +1000, David Gibson wrote:
> > 
> > Now, this is an ICS subclass, so why shouldn't it directly poke at the
> > target ICP ?
> 
> That's ok in theory, but causing it to expose the icp interface to a
> new module isn't great.
> 
> > It's an alternate to the normal ICS since it behaves a bit
> > differently (PQ bits & all).
> 
> AFAICT the PQ bits are effectively another filtering layer on top of
> the same ICS logic as elsewhere.  I think it's better if we can share
> that shared logic, rather than replicating it.

I don't know, is there much shared logic ? And the shared bits are the
subclassing, that's handled that way...

This is really a different piece of HW, a separate ICS implementation,
that has its own quirks, is configured via different registers, does
EOI differently etc...

Even the resend stuff is done differently, the resend bitmap is
actually SW visible via an IODA table.

I mean, we could (maybe we do these days not sure) have an ICS
superclass wrapper on the actual icp_irq() but that's really all there
is to it I think.

Cheers,
Ben.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-23 21:37     ` Cédric Le Goater
@ 2018-07-24  1:29       ` David Gibson
  0 siblings, 0 replies; 29+ messages in thread
From: David Gibson @ 2018-07-24  1:29 UTC (permalink / raw)
  To: Cédric Le Goater
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

On Mon, Jul 23, 2018 at 11:37:06PM +0200, Cédric Le Goater wrote:
> On 07/18/2018 08:12 AM, David Gibson wrote:
> >> +static void pnv_phb3_get_phb_id(Object *obj, Visitor *v, const char *name,
> >> +                              void *opaque, Error **errp)
> >> +{
> >> +    Property *prop = opaque;
> >> +    uint32_t *ptr = qdev_get_prop_ptr(DEVICE(obj), prop);
> >> +
> >> +    visit_type_uint32(v, name, ptr, errp);
> >> +}
> >> +
> >> +static void pnv_phb3_set_phb_id(Object *obj, Visitor *v, const char *name,
> >> +                              void *opaque, Error **errp)
> >> +{
> >> +    PnvPHB3 *phb = PNV_PHB3(obj);
> >> +    uint32_t phb_id;
> >> +    Error *local_err = NULL;
> >> +
> >> +    visit_type_uint32(v, name, &phb_id, &local_err);
> >> +    if (local_err) {
> >> +        error_propagate(errp, local_err);
> >> +        return;
> >> +    }
> >> +
> >> +    /*
> >> +     * Limit to a maximum of 6 PHBs per chip
> >> +     */
> >> +    if (phb_id >= PNV8_CHIP_PHB3_MAX) {
> >> +        error_setg(errp, "invalid PHB index: '%d'", phb_id);
> >> +        return;
> >> +    }
> >> +
> >> +    phb->phb_id = phb_id;
> >> +}
> >> +
> >> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
> >> +    .name = "irq",
> >> +    .get = pnv_phb3_get_phb_id,
> >> +    .set = pnv_phb3_set_phb_id,
> >> +};
> > Can't you use a static DeviceProps style property for this, which is a
> > bit simpler?
> 
> yes but we will need these ops for user creatable devices. This is where
> we will check the chip id.

Why?  Can't we check that during realize()?  That's a common pattern
for static properties.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-23 23:55         ` Benjamin Herrenschmidt
@ 2018-07-24  2:14           ` David Gibson
  2018-07-24  3:49             ` Benjamin Herrenschmidt
  0 siblings, 1 reply; 29+ messages in thread
From: David Gibson @ 2018-07-24  2:14 UTC (permalink / raw)
  To: Benjamin Herrenschmidt
  Cc: Cédric Le Goater, qemu-ppc, qemu-devel, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

[-- Attachment #1: Type: text/plain, Size: 1637 bytes --]

On Tue, Jul 24, 2018 at 09:55:53AM +1000, Benjamin Herrenschmidt wrote:
> On Mon, 2018-07-23 at 14:16 +1000, David Gibson wrote:
> > > 
> > > Now, this is an ICS subclass, so why shouldn't it directly poke at the
> > > target ICP ?
> > 
> > That's ok in theory, but causing it to expose the icp interface to a
> > new module isn't great.
> > 
> > > It's an alternate to the normal ICS since it behaves a bit
> > > differently (PQ bits & all).
> > 
> > AFAICT the PQ bits are effectively another filtering layer on top of
> > the same ICS logic as elsewhere.  I think it's better if we can share
> > that shared logic, rather than replicating it.
> 
> I don't know, is there much shared logic ? And the shared bits are the
> subclassing, that's handled that way...
> 
> This is really a different piece of HW, a separate ICS implementation,
> that has its own quirks, is configured via different registers, does
> EOI differently etc...
> 
> Even the resend stuff is done differently, the resend bitmap is
> actually SW visible via an IODA table.
> 
> I mean, we could (maybe we do these days not sure) have an ICS
> superclass wrapper on the actual icp_irq() but that's really all there
> is to it I think.

Hm, yeah, fair enough.  I realized what I was suggesting would
actually need another layer of qirqs than we have, so it's not a good
idea after all.  Ok, I'm happy proceeding with exposing icp_irq().

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-24  2:14           ` David Gibson
@ 2018-07-24  3:49             ` Benjamin Herrenschmidt
  2018-07-24  4:10               ` David Gibson
  0 siblings, 1 reply; 29+ messages in thread
From: Benjamin Herrenschmidt @ 2018-07-24  3:49 UTC (permalink / raw)
  To: David Gibson
  Cc: Cédric Le Goater, qemu-ppc, qemu-devel, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

On Tue, 2018-07-24 at 12:14 +1000, David Gibson wrote:
> > I don't know, is there much shared logic ? And the shared bits are the
> > subclassing, that's handled that way...
> > 
> > This is really a different piece of HW, a separate ICS implementation,
> > that has its own quirks, is configured via different registers, does
> > EOI differently etc...
> > 
> > Even the resend stuff is done differently, the resend bitmap is
> > actually SW visible via an IODA table.
> > 
> > I mean, we could (maybe we do these days not sure) have an ICS
> > superclass wrapper on the actual icp_irq() but that's really all there
> > is to it I think.
> 
> Hm, yeah, fair enough.  I realized what I was suggesting would
> actually need another layer of qirqs than we have, so it's not a good
> idea after all.  Ok, I'm happy proceeding with exposing icp_irq().

The only idea I have if you want to keep icp_* private is to put a
'helper' in the ICS superclass that the subclass uses to encapsulate
the ICP call, but at this point it's just churn.

But yeah you really don't want a layer of qirqs, it wouldn't work any
way, it's really an ICS, it will send messages to the ICP with a XIRR
value in them etc... just like an ICS, it's not somethign you want
qirq's for .

Cheers,
Ben.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-24  3:49             ` Benjamin Herrenschmidt
@ 2018-07-24  4:10               ` David Gibson
  0 siblings, 0 replies; 29+ messages in thread
From: David Gibson @ 2018-07-24  4:10 UTC (permalink / raw)
  To: Benjamin Herrenschmidt
  Cc: Cédric Le Goater, qemu-ppc, qemu-devel, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

On Tue, Jul 24, 2018 at 01:49:32PM +1000, Benjamin Herrenschmidt wrote:
> On Tue, 2018-07-24 at 12:14 +1000, David Gibson wrote:
> > > I don't know, is there much shared logic ? And the shared bits are the
> > > subclassing, that's handled that way...
> > > 
> > > This is really a different piece of HW, a separate ICS implementation,
> > > that has its own quirks, is configured via different registers, does
> > > EOI differently etc...
> > > 
> > > Even the resend stuff is done differently, the resend bitmap is
> > > actually SW visible via an IODA table.
> > > 
> > > I mean, we could (maybe we do these days not sure) have an ICS
> > > superclass wrapper on the actual icp_irq() but that's really all there
> > > is to it I think.
> > 
> > Hm, yeah, fair enough.  I realized what I was suggesting would
> > actually need another layer of qirqs than we have, so it's not a good
> > idea after all.  Ok, I'm happy proceeding with exposing icp_irq().
> 
> The only idea I have if you want to keep icp_* private is to put a
> 'helper' in the ICS superclass that the subclass uses to encapsulate
> the ICP call, but at this point it's just churn.

I concur.

> 
> But yeah you really don't want a layer of qirqs, it wouldn't work any
> way, it's really an ICS, it will send messages to the ICP with a XIRR
> value in them etc... just like an ICS, it's not somethign you want
> qirq's for .
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-18  6:12   ` David Gibson
                       ` (3 preceding siblings ...)
  2018-07-23 21:38     ` Cédric Le Goater
@ 2018-07-24 12:18     ` Cédric Le Goater
  2018-07-27  5:32       ` David Gibson
  4 siblings, 1 reply; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-24 12:18 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

Hello,

>From the discussion on the XICS MSI object, I gather that exporting
icp_irq is fine. 

Some more comments below, I have tried to answer the parts that were 
not addressed yet. 

[ ... ]

>> +/*
>> + * The CONFIG_DATA register expects little endian accesses, but as the
>> + * region is big endian, we have to swap the value.
>> + */
>> +static void pnv_phb3_config_write(PnvPHB3 *phb, unsigned off,
>> +                                  unsigned size, uint64_t val)
>> +{
>> +    uint32_t cfg_addr, limit;
>> +    PCIDevice *pdev;
>> +
>> +    pdev = pnv_phb3_find_cfg_dev(phb);
>> +    if (!pdev) {
>> +        return;
>> +    }
>> +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xfff;
>> +    cfg_addr |= off;
> 
> This looks weird - there are callers of this that appear to have low
> bits in 'off', then you're ORing it with overlapping low bits.

Here is an extract of the specs of the PHB_CONFIG_ADDRESS register :

  20:29 Register Number(00:09) RW 0

  Register Number to use for the configuration cycle access. 

  30:31 Register Number(10:11) RO 0

  Don’t Cares. The value written will be dropped, and zeros are
  always returned for a read.  

So, yes, we shoud be using 0xffc, just like in read.

>> +    limit = pci_config_size(pdev);
>> +    if (limit <= cfg_addr) {
>> +        /* conventional pci device can be behind pcie-to-pci bridge.
>> +           256 <= addr < 4K has no effects. */
>> +        return;
>> +    }
>> +    switch (size) {
>> +    case 1:
>> +        break;
>> +    case 2:
>> +        val = bswap16(val);
>> +        break;
>> +    case 4:
>> +        val = bswap32(val);
>> +        break;
>> +    default:
>> +        g_assert_not_reached();
>> +    }
>> +    pci_host_config_write_common(pdev, cfg_addr, limit, val, size);
>> +}
>> +
>> +static uint64_t pnv_phb3_config_read(PnvPHB3 *phb, unsigned off,
>> +                                     unsigned size)
>> +{
>> +    uint32_t cfg_addr, limit;
>> +    PCIDevice *pdev;
>> +    uint64_t val;
>> +
>> +    pdev = pnv_phb3_find_cfg_dev(phb);
>> +    if (!pdev) {
>> +        return ~0ull;
>> +    }
>> +    cfg_addr = (phb->regs[PHB_CONFIG_ADDRESS >> 3] >> 32) & 0xffc;
>> +    cfg_addr |= off;
> 
> This looks better, should it be 0xffc in the write path as well?

yes.

>> +    limit = pci_config_size(pdev);
>> +    if (limit <= cfg_addr) {
>> +        /* conventional pci device can be behind pcie-to-pci bridge.
>> +           256 <= addr < 4K has no effects. */
>> +        return ~0ull;
>> +    }
>> +    val = pci_host_config_read_common(pdev, cfg_addr, limit, size);
>> +    switch (size) {
>> +    case 1:
>> +        return val;
>> +    case 2:
>> +        return bswap16(val);
>> +    case 4:
>> +        return bswap32(val);
>> +    default:
>> +        g_assert_not_reached();
>> +    }
>> +}
>> +
>> +static void pnv_phb3_check_m32(PnvPHB3 *phb)
>> +{
>> +    uint64_t base, start, size;
>> +    MemoryRegion *parent;
>> +    PnvPBCQState *pbcq = &phb->pbcq;
>> +
>> +    if (phb->m32_mapped) {
>> +        memory_region_del_subregion(phb->mr_m32.container, &phb->mr_m32);
>> +        phb->m32_mapped = false;
> 
> Could you use memory_region_set_enabled() rather than having to add
> and delete the subregion and keep track of it in this separate
> variable?

pnv_phb3_check_m32/m64 remap the regions. 

[ ... ]

>> +    case IODA2_TBL_PEEV:
>> +        tptr = phb->ioda_PEEV;
>> +        mask = 3;
>> +        break;
>> +    default:
>> +        return NULL;
>> +    }
>> +    index &= mask;
> 
> Do we want to silently mask, rather than logging an error if there's
> an access out of bounds for a particular table?

I will add an error.

[ ... ]

>> +    /* Handle masking */
>> +    switch (off) {
>> +    case PHB_M64_UPPER_BITS:
> 
> Couldn't you handle this in the switch below - it should be ok to
> momentarily have the invalid bits set in your reg case, as long as you
> mask them before applying the side-effects, yes?

yes.

> That said... shouldn't you filter our invalid or read-only regs before
> updating the cache?

hmm, the current model relies on the fact that some registers have
their value implicitly updated. But I guess, we can dump which are
accessed and implement a default behavior for these. I will take a
look.

[ ... ]

>> +static const MemoryRegionOps pnv_phb3_msi_ops = {
>> +    /* There is no .read as the read result is undefined by PCI spec */
> 
> What will qemu do if it hits a NULL read here?  The behaviour may be
> undefind, but we'd prefer it didn't crash qemu..

I will add one with a standard error message. It is better to have
some output anyway.

[ ... ]

>> +static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
>> +{
>> +    PCIHostState *pcih = PCI_HOST_BRIDGE(phb);
>> +    PCIDevice *brdev;
>> +    PCIDevice *pdev;
>> +    PCIBus *parent;
>> +    uint8_t chassis = phb->chip_id * 4 + phb->phb_id;
>> +    uint8_t chassis_nr = 128;
>> +
>> +    /* Add root complex */
>> +    pdev = pci_create(pcih->bus, 0, TYPE_PNV_PHB3_RC);
>> +    object_property_add_child(OBJECT(phb), "phb3-rc", OBJECT(pdev), NULL);
>> +    qdev_prop_set_uint8(DEVICE(pdev), "chassis", chassis);
>> +    qdev_prop_set_uint16(DEVICE(pdev), "slot", 1);
>> +    qdev_init_nofail(DEVICE(pdev));
>> +
>> +    /* Setup bus for that chip */
>> +    parent = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
>> +
>> +    brdev = pci_create(parent, 0, "pci-bridge");
> 
> What is this pci bridge representing?  I know PCI-e PHBs typically
> have a pseudo P2P bridge right under them, but isn't that represnted
> by the root complex above?

This is the legacy pci bridge under the pcie bus. 

Here is is the qdev hierarchy :

  dev: pnv-phb3, id ""
    index = 0 (0x0)
    chip-id = 0 (0x0)
    bus: phb3-root-bus
      type pnv-phb3-root-bus
      dev: pnv-phb3-rc, id ""
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 1 (0x1)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        x-pcie-lnksta-dllla = true
        x-pcie-extcap-init = true
        class PCI bridge, addr 00:00.0, pci id 1014:03dc (sub 0000:0000)
        bus: pcie.0
          type PCIE
          dev: pci-bridge, id ""
            chassis_nr = 128 (0x80)
            msi = "off"
            shpc = false
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            x-pcie-lnksta-dllla = true
            x-pcie-extcap-init = true
            class PCI bridge, addr 00:00.0, pci id 1b36:0001 (sub 0000:0000)
            bus: pci.0
              type PCI

[ ... ]

>> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
>> +    .name = "irq",
>> +    .get = pnv_phb3_get_phb_id,
>> +    .set = pnv_phb3_set_phb_id,
>> +};
> 
> Can't you use a static DeviceProps style property for this, which is a
> bit simpler?

OK. We will address user creatable PHBs in some other way. Most
certainly in the realize routine like you suggested.

Thanks,

C. 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 1/2] " Cédric Le Goater
  2018-07-09  7:22   ` Cédric Le Goater
  2018-07-18  6:12   ` David Gibson
@ 2018-07-26  9:03   ` Cédric Le Goater
  2018-07-26 22:36     ` Benjamin Herrenschmidt
  2 siblings, 1 reply; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-26  9:03 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

Ben,

I have found out recently that the QEMU PowerNV could hang while accessing
the disk. 

The issue seems to be the phb3_msi_try_send() routine when called from 
the resend() handler. The 'P' is ignored in that case but not the 'Q' 
bit which means that no interrupt will be resent if P|Q are set. 

See the log extract below  :

  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005000000fd eff pq=0 prio=5 server=8 ignore_p=0
  PHB3(phb3_msi_set_p): MSI 0: setting P
  PHB3(phb3_msi_set_p):  IVE readback: 0x2005010000fd
  PHB3(phb3_msi_reject): MSI 0 rejected
  PHB3(phb3_msi_resend): MSI resend...
  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010000fd eff pq=0 prio=5 server=8 ignore_p=1
  PHB3(phb3_msi_set_p): MSI 0: setting P
  PHB3(phb3_msi_set_p):  IVE readback: 0x2005010000fd
  PHB3(phb3_msi_reject): MSI 0 rejected
  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010000fd eff pq=2 prio=5 server=8 ignore_p=0
  PHB3(phb3_msi_set_q): MSI 0: setting Q
  PHB3(phb3_msi_set_q):  IVE readback: 0x2005010100fd
  PHB3(phb3_msi_resend): MSI resend...
  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=1 prio=5 server=8 ignore_p=1
  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
  PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
  ... goes on and on ...
  hangs

I have added the relevant code at the bottom of the email. 


If the 'Q' bit is ignored also, the results are good with a SATA drive 
or a SCSI drive using the megasas model. Do you think this is correct ?
I would say so but I am still discovering that part.

I have no idea why it didn't show up before. May be because we mostly 
used virtio-blk.

Thanks,

C. 

> +static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
> +{
> +    ICSState *ics = ICS_BASE(msi);
> +    uint64_t ive;
> +    uint64_t server, prio, pq, gen;
> +
> +    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
> +        return;
> +    }
> +
> +    server = GETFIELD(IODA2_IVT_SERVER, ive);
> +    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
> +    pq = GETFIELD(IODA2_IVT_Q, ive);
> +    if (!ignore_p) {
> +        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
> +    }
> +    gen = GETFIELD(IODA2_IVT_GEN, ive);
> +
> +    /*
> +     * The low order 2 bits are the link pointer (Type II interrupts).
> +     * Shift back to get a valid IRQ server.
> +     */
> +    server >>= 2;
> +
> +    switch (pq) {
> +    case 0: /* 00 */
> +        if (prio == 0xff) {
> +            /* Masked, set Q */
> +            phb3_msi_set_q(msi, srcno);
> +        } else {
> +            /* Enabled, set P and send */
> +            phb3_msi_set_p(msi, srcno, gen);
> +            icp_irq(ics, server, srcno + ics->offset, prio);
> +        }
> +        break;
> +    case 2: /* 10 */
> +        /* Already pending, set Q */
> +        phb3_msi_set_q(msi, srcno);
> +        break;
> +    case 1: /* 01 */
> +    case 3: /* 11 */
> +    default:
> +        /* Just drop stuff if Q already set */
> +        break;
> +    }
> +}
> +
> +static void phb3_msi_set_irq(void *opaque, int srcno, int val)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(opaque);
> +
> +    if (val) {
> +        phb3_msi_try_send(msi, srcno, false);
> +    }
> +}

[ ... ]
 
> +static void phb3_msi_resend(ICSState *ics)
> +{
> +    Phb3MsiState *msi = PHB3_MSI(ics);
> +    unsigned int i, j;
> +
> +    if (msi->rba_sum == 0) {
> +        return;
> +    }
> +
> +    for (i = 0; i < 32; i++) {
> +        if ((msi->rba_sum & (1u << i)) == 0) {
> +            continue;
> +        }
> +        msi->rba_sum &= ~(1u << i);
> +        for (j = 0; j < 64; j++) {
> +            if ((msi->rba[i] & (1ull << j)) == 0) {
> +                continue;
> +            }
> +            msi->rba[i] &= ~(1u << j);
> +            phb3_msi_try_send(msi, i * 64 + j, true);
> +        }
> +    }
> +}

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-26  9:03   ` Cédric Le Goater
@ 2018-07-26 22:36     ` Benjamin Herrenschmidt
  2018-07-27  7:16       ` Cédric Le Goater
  0 siblings, 1 reply; 29+ messages in thread
From: Benjamin Herrenschmidt @ 2018-07-26 22:36 UTC (permalink / raw)
  To: Cédric Le Goater, David Gibson
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On Thu, 2018-07-26 at 11:03 +0200, Cédric Le Goater wrote:
> Ben,
> 
> I have found out recently that the QEMU PowerNV could hang while accessing
> the disk. 
> 
> The issue seems to be the phb3_msi_try_send() routine when called from 
> the resend() handler. The 'P' is ignored in that case but not the 'Q' 
> bit which means that no interrupt will be resent if P|Q are set. 

I'd have to remember how PQ works on P8 ... my gut feeling is that we
should resend if P=1 but I'm no 100% certain.

> See the log extract below  :
> 
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005000000fd eff pq=0 prio=5 server=8 ignore_p=0
>   PHB3(phb3_msi_set_p): MSI 0: setting P
>   PHB3(phb3_msi_set_p):  IVE readback: 0x2005010000fd
>   PHB3(phb3_msi_reject): MSI 0 rejected
>   PHB3(phb3_msi_resend): MSI resend...
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010000fd eff pq=0 prio=5 server=8 ignore_p=1
>   PHB3(phb3_msi_set_p): MSI 0: setting P
>   PHB3(phb3_msi_set_p):  IVE readback: 0x2005010000fd
>   PHB3(phb3_msi_reject): MSI 0 rejected
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010000fd eff pq=2 prio=5 server=8 ignore_p=0
>   PHB3(phb3_msi_set_q): MSI 0: setting Q
>   PHB3(phb3_msi_set_q):  IVE readback: 0x2005010100fd
>   PHB3(phb3_msi_resend): MSI resend...
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=1 prio=5 server=8 ignore_p=1
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>   ... goes on and on ...
>   hangs
> 
> I have added the relevant code at the bottom of the email. 
> 
> 
> If the 'Q' bit is ignored also, the results are good with a SATA drive 
> or a SCSI drive using the megasas model. Do you think this is correct ?
> I would say so but I am still discovering that part.
> 
> I have no idea why it didn't show up before. May be because we mostly 
> used virtio-blk.
> 
> Thanks,
> 
> C. 
> 
> > +static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
> > +{
> > +    ICSState *ics = ICS_BASE(msi);
> > +    uint64_t ive;
> > +    uint64_t server, prio, pq, gen;
> > +
> > +    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
> > +        return;
> > +    }
> > +
> > +    server = GETFIELD(IODA2_IVT_SERVER, ive);
> > +    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
> > +    pq = GETFIELD(IODA2_IVT_Q, ive);
> > +    if (!ignore_p) {
> > +        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
> > +    }
> > +    gen = GETFIELD(IODA2_IVT_GEN, ive);
> > +
> > +    /*
> > +     * The low order 2 bits are the link pointer (Type II interrupts).
> > +     * Shift back to get a valid IRQ server.
> > +     */
> > +    server >>= 2;
> > +
> > +    switch (pq) {
> > +    case 0: /* 00 */
> > +        if (prio == 0xff) {
> > +            /* Masked, set Q */
> > +            phb3_msi_set_q(msi, srcno);
> > +        } else {
> > +            /* Enabled, set P and send */
> > +            phb3_msi_set_p(msi, srcno, gen);
> > +            icp_irq(ics, server, srcno + ics->offset, prio);
> > +        }
> > +        break;
> > +    case 2: /* 10 */
> > +        /* Already pending, set Q */
> > +        phb3_msi_set_q(msi, srcno);
> > +        break;
> > +    case 1: /* 01 */
> > +    case 3: /* 11 */
> > +    default:
> > +        /* Just drop stuff if Q already set */
> > +        break;
> > +    }
> > +}
> > +
> > +static void phb3_msi_set_irq(void *opaque, int srcno, int val)
> > +{
> > +    Phb3MsiState *msi = PHB3_MSI(opaque);
> > +
> > +    if (val) {
> > +        phb3_msi_try_send(msi, srcno, false);
> > +    }
> > +}
> 
> [ ... ]
>  
> > +static void phb3_msi_resend(ICSState *ics)
> > +{
> > +    Phb3MsiState *msi = PHB3_MSI(ics);
> > +    unsigned int i, j;
> > +
> > +    if (msi->rba_sum == 0) {
> > +        return;
> > +    }
> > +
> > +    for (i = 0; i < 32; i++) {
> > +        if ((msi->rba_sum & (1u << i)) == 0) {
> > +            continue;
> > +        }
> > +        msi->rba_sum &= ~(1u << i);
> > +        for (j = 0; j < 64; j++) {
> > +            if ((msi->rba[i] & (1ull << j)) == 0) {
> > +                continue;
> > +            }
> > +            msi->rba[i] &= ~(1u << j);
> > +            phb3_msi_try_send(msi, i * 64 + j, true);
> > +        }
> > +    }
> > +}
> 
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-24 12:18     ` Cédric Le Goater
@ 2018-07-27  5:32       ` David Gibson
  2018-07-27  7:33         ` Cédric Le Goater
  2018-07-27  8:08         ` Benjamin Herrenschmidt
  0 siblings, 2 replies; 29+ messages in thread
From: David Gibson @ 2018-07-27  5:32 UTC (permalink / raw)
  To: Cédric Le Goater
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

[-- Attachment #1: Type: text/plain, Size: 4362 bytes --]

On Tue, Jul 24, 2018 at 02:18:30PM +0200, Cédric Le Goater wrote:
> Hello,
> 
> >From the discussion on the XICS MSI object, I gather that exporting
> icp_irq is fine. 
> 
> Some more comments below, I have tried to answer the parts that were 
> not addressed yet.

[snip]
> > Could you use memory_region_set_enabled() rather than having to add
> > and delete the subregion and keep track of it in this separate
> > variable?
> 
> pnv_phb3_check_m32/m64 remap the regions.

Ah, ok.

[snip]
> > That said... shouldn't you filter our invalid or read-only regs before
> > updating the cache?
> 
> hmm, the current model relies on the fact that some registers have
> their value implicitly updated. But I guess, we can dump which are
> accessed and implement a default behavior for these. I will take a
> look.

I don't really follow what you're saying here, but I guess I'll see
what it looks like in te next spin.

> 
> [ ... ]
> 
> >> +static const MemoryRegionOps pnv_phb3_msi_ops = {
> >> +    /* There is no .read as the read result is undefined by PCI spec */
> > 
> > What will qemu do if it hits a NULL read here?  The behaviour may be
> > undefind, but we'd prefer it didn't crash qemu..
> 
> I will add one with a standard error message. It is better to have
> some output anyway.
> 
> [ ... ]
> 
> >> +static void pnv_phb3_pci_create(PnvPHB3 *phb, Error **errp)
> >> +{
> >> +    PCIHostState *pcih = PCI_HOST_BRIDGE(phb);
> >> +    PCIDevice *brdev;
> >> +    PCIDevice *pdev;
> >> +    PCIBus *parent;
> >> +    uint8_t chassis = phb->chip_id * 4 + phb->phb_id;
> >> +    uint8_t chassis_nr = 128;
> >> +
> >> +    /* Add root complex */
> >> +    pdev = pci_create(pcih->bus, 0, TYPE_PNV_PHB3_RC);
> >> +    object_property_add_child(OBJECT(phb), "phb3-rc", OBJECT(pdev), NULL);
> >> +    qdev_prop_set_uint8(DEVICE(pdev), "chassis", chassis);
> >> +    qdev_prop_set_uint16(DEVICE(pdev), "slot", 1);
> >> +    qdev_init_nofail(DEVICE(pdev));
> >> +
> >> +    /* Setup bus for that chip */
> >> +    parent = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
> >> +
> >> +    brdev = pci_create(parent, 0, "pci-bridge");
> > 
> > What is this pci bridge representing?  I know PCI-e PHBs typically
> > have a pseudo P2P bridge right under them, but isn't that represnted
> > by the root complex above?
> 
> This is the legacy pci bridge under the pcie bus.

Ah, ok.  Didn't realise there was a vanilla PCI bridge built in.

> Here is is the qdev hierarchy :
> 
>   dev: pnv-phb3, id ""
>     index = 0 (0x0)
>     chip-id = 0 (0x0)
>     bus: phb3-root-bus
>       type pnv-phb3-root-bus
>       dev: pnv-phb3-rc, id ""
>         power_controller_present = true
>         chassis = 0 (0x0)
>         slot = 1 (0x1)
>         port = 0 (0x0)
>         aer_log_max = 8 (0x8)
>         addr = 00.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         x-pcie-lnksta-dllla = true
>         x-pcie-extcap-init = true
>         class PCI bridge, addr 00:00.0, pci id 1014:03dc (sub 0000:0000)
>         bus: pcie.0
>           type PCIE
>           dev: pci-bridge, id ""
>             chassis_nr = 128 (0x80)
>             msi = "off"
>             shpc = false
>             addr = 00.0
>             romfile = ""
>             rombar = 1 (0x1)
>             multifunction = false
>             command_serr_enable = true
>             x-pcie-lnksta-dllla = true
>             x-pcie-extcap-init = true
>             class PCI bridge, addr 00:00.0, pci id 1b36:0001 (sub 0000:0000)
>             bus: pci.0
>               type PCI
> 
> [ ... ]
> 
> >> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
> >> +    .name = "irq",
> >> +    .get = pnv_phb3_get_phb_id,
> >> +    .set = pnv_phb3_set_phb_id,
> >> +};
> > 
> > Can't you use a static DeviceProps style property for this, which is a
> > bit simpler?
> 
> OK. We will address user creatable PHBs in some other way. Most
> certainly in the realize routine like you suggested.
> 
> Thanks,
> 
> C. 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-26 22:36     ` Benjamin Herrenschmidt
@ 2018-07-27  7:16       ` Cédric Le Goater
  2018-07-27  8:18         ` Benjamin Herrenschmidt
  0 siblings, 1 reply; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-27  7:16 UTC (permalink / raw)
  To: Benjamin Herrenschmidt, David Gibson
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On 07/27/2018 12:36 AM, Benjamin Herrenschmidt wrote:
> On Thu, 2018-07-26 at 11:03 +0200, Cédric Le Goater wrote:
>> Ben,
>>
>> I have found out recently that the QEMU PowerNV could hang while accessing
>> the disk. 
>>
>> The issue seems to be the phb3_msi_try_send() routine when called from 
>> the resend() handler. The 'P' is ignored in that case but not the 'Q' 
>> bit which means that no interrupt will be resent if P|Q are set. 
> 
> I'd have to remember how PQ works on P8 ... my gut feeling is that we
> should resend if P=1 but I'm no 100% certain.

This is not exactly what the code does. To force a resend, it ignores 
P but if Q=1, it bails out without doing anything, like a normal trigger 
would do. So I think that in the resend case we should ignore Q as well.

Thanks,

C.


> 
>> See the log extract below  :
>>
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005000000fd eff pq=0 prio=5 server=8 ignore_p=0
>>   PHB3(phb3_msi_set_p): MSI 0: setting P
>>   PHB3(phb3_msi_set_p):  IVE readback: 0x2005010000fd
>>   PHB3(phb3_msi_reject): MSI 0 rejected
>>   PHB3(phb3_msi_resend): MSI resend...
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010000fd eff pq=0 prio=5 server=8 ignore_p=1
>>   PHB3(phb3_msi_set_p): MSI 0: setting P
>>   PHB3(phb3_msi_set_p):  IVE readback: 0x2005010000fd
>>   PHB3(phb3_msi_reject): MSI 0 rejected
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010000fd eff pq=2 prio=5 server=8 ignore_p=0
>>   PHB3(phb3_msi_set_q): MSI 0: setting Q
>>   PHB3(phb3_msi_set_q):  IVE readback: 0x2005010100fd
>>   PHB3(phb3_msi_resend): MSI resend...
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=1 prio=5 server=8 ignore_p=1
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>>   PHB3(phb3_msi_try_send): MSI 0: try_send, ive=0x2005010100fd eff pq=3 prio=5 server=8 ignore_p=0
>>   ... goes on and on ...
>>   hangs
>>
>> I have added the relevant code at the bottom of the email. 
>>
>>
>> If the 'Q' bit is ignored also, the results are good with a SATA drive 
>> or a SCSI drive using the megasas model. Do you think this is correct ?
>> I would say so but I am still discovering that part.
>>
>> I have no idea why it didn't show up before. May be because we mostly 
>> used virtio-blk.
>>
>> Thanks,
>>
>> C. 
>>
>>> +static void phb3_msi_try_send(Phb3MsiState *msi, int srcno, bool ignore_p)
>>> +{
>>> +    ICSState *ics = ICS_BASE(msi);
>>> +    uint64_t ive;
>>> +    uint64_t server, prio, pq, gen;
>>> +
>>> +    if (!phb3_msi_read_ive(msi->phb, srcno, &ive)) {
>>> +        return;
>>> +    }
>>> +
>>> +    server = GETFIELD(IODA2_IVT_SERVER, ive);
>>> +    prio = GETFIELD(IODA2_IVT_PRIORITY, ive);
>>> +    pq = GETFIELD(IODA2_IVT_Q, ive);
>>> +    if (!ignore_p) {
>>> +        pq |= GETFIELD(IODA2_IVT_P, ive) << 1;
>>> +    }
>>> +    gen = GETFIELD(IODA2_IVT_GEN, ive);
>>> +
>>> +    /*
>>> +     * The low order 2 bits are the link pointer (Type II interrupts).
>>> +     * Shift back to get a valid IRQ server.
>>> +     */
>>> +    server >>= 2;
>>> +
>>> +    switch (pq) {
>>> +    case 0: /* 00 */
>>> +        if (prio == 0xff) {
>>> +            /* Masked, set Q */
>>> +            phb3_msi_set_q(msi, srcno);
>>> +        } else {
>>> +            /* Enabled, set P and send */
>>> +            phb3_msi_set_p(msi, srcno, gen);
>>> +            icp_irq(ics, server, srcno + ics->offset, prio);
>>> +        }
>>> +        break;
>>> +    case 2: /* 10 */
>>> +        /* Already pending, set Q */
>>> +        phb3_msi_set_q(msi, srcno);
>>> +        break;
>>> +    case 1: /* 01 */
>>> +    case 3: /* 11 */
>>> +    default:
>>> +        /* Just drop stuff if Q already set */
>>> +        break;
>>> +    }
>>> +}
>>> +
>>> +static void phb3_msi_set_irq(void *opaque, int srcno, int val)
>>> +{
>>> +    Phb3MsiState *msi = PHB3_MSI(opaque);
>>> +
>>> +    if (val) {
>>> +        phb3_msi_try_send(msi, srcno, false);
>>> +    }
>>> +}
>>
>> [ ... ]
>>  
>>> +static void phb3_msi_resend(ICSState *ics)
>>> +{
>>> +    Phb3MsiState *msi = PHB3_MSI(ics);
>>> +    unsigned int i, j;
>>> +
>>> +    if (msi->rba_sum == 0) {
>>> +        return;
>>> +    }
>>> +
>>> +    for (i = 0; i < 32; i++) {
>>> +        if ((msi->rba_sum & (1u << i)) == 0) {
>>> +            continue;
>>> +        }
>>> +        msi->rba_sum &= ~(1u << i);
>>> +        for (j = 0; j < 64; j++) {
>>> +            if ((msi->rba[i] & (1ull << j)) == 0) {
>>> +                continue;
>>> +            }
>>> +            msi->rba[i] &= ~(1u << j);
>>> +            phb3_msi_try_send(msi, i * 64 + j, true);
>>> +        }
>>> +    }
>>> +}
>>
>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-27  5:32       ` David Gibson
@ 2018-07-27  7:33         ` Cédric Le Goater
  2018-07-27  8:08         ` Benjamin Herrenschmidt
  1 sibling, 0 replies; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-27  7:33 UTC (permalink / raw)
  To: David Gibson
  Cc: qemu-ppc, qemu-devel, Benjamin Herrenschmidt, Marcel Apfelbaum,
	Andrea Bolognani, Michael S. Tsirkin

>>> That said... shouldn't you filter our invalid or read-only regs before
>>> updating the cache?
>>
>> hmm, the current model relies on the fact that some registers have
>> their value implicitly updated. But I guess, we can dump which are
>> accessed and implement a default behavior for these. I will take a
>> look.
> 
> I don't really follow what you're saying here, but I guess I'll see
> what it looks like in te next spin.

My idea was to printf the registers being used and populate the switch 
statement, but there are too many of them and it would only pollute 
the code for not much benefit I think 

I will keep it as it is and may be fix a few things if possible.

C.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-27  5:32       ` David Gibson
  2018-07-27  7:33         ` Cédric Le Goater
@ 2018-07-27  8:08         ` Benjamin Herrenschmidt
  2018-07-27  8:25           ` Cédric Le Goater
  1 sibling, 1 reply; 29+ messages in thread
From: Benjamin Herrenschmidt @ 2018-07-27  8:08 UTC (permalink / raw)
  To: David Gibson, Cédric Le Goater
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On Fri, 2018-07-27 at 15:32 +1000, David Gibson wrote:
> 
> > > What is this pci bridge representing?  I know PCI-e PHBs typically
> > > have a pseudo P2P bridge right under them, but isn't that represnted
> > > by the root complex above?
> > 
> > This is the legacy pci bridge under the pcie bus.
> 
> Ah, ok.  Didn't realise there was a vanilla PCI bridge built in.

There isn't. That should probably be created by the machine.

> > Here is is the qdev hierarchy :
> > 
> >   dev: pnv-phb3, id ""
> >     index = 0 (0x0)
> >     chip-id = 0 (0x0)
> >     bus: phb3-root-bus
> >       type pnv-phb3-root-bus
> >       dev: pnv-phb3-rc, id ""
> >         power_controller_present = true
> >         chassis = 0 (0x0)
> >         slot = 1 (0x1)
> >         port = 0 (0x0)
> >         aer_log_max = 8 (0x8)
> >         addr = 00.0
> >         romfile = ""
> >         rombar = 1 (0x1)
> >         multifunction = false
> >         command_serr_enable = true
> >         x-pcie-lnksta-dllla = true
> >         x-pcie-extcap-init = true
> >         class PCI bridge, addr 00:00.0, pci id 1014:03dc (sub 0000:0000)
> >         bus: pcie.0
> >           type PCIE
> >           dev: pci-bridge, id ""
> >             chassis_nr = 128 (0x80)
> >             msi = "off"
> >             shpc = false
> >             addr = 00.0
> >             romfile = ""
> >             rombar = 1 (0x1)
> >             multifunction = false
> >             command_serr_enable = true
> >             x-pcie-lnksta-dllla = true
> >             x-pcie-extcap-init = true
> >             class PCI bridge, addr 00:00.0, pci id 1b36:0001 (sub 0000:0000)
> >             bus: pci.0
> >               type PCI
> > 
> > [ ... ]
> > 
> > >> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
> > >> +    .name = "irq",
> > >> +    .get = pnv_phb3_get_phb_id,
> > >> +    .set = pnv_phb3_set_phb_id,
> > >> +};
> > > 
> > > Can't you use a static DeviceProps style property for this, which is a
> > > bit simpler?
> > 
> > OK. We will address user creatable PHBs in some other way. Most
> > certainly in the realize routine like you suggested.
> > 
> > Thanks,
> > 
> > C. 
> > 
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-27  7:16       ` Cédric Le Goater
@ 2018-07-27  8:18         ` Benjamin Herrenschmidt
  0 siblings, 0 replies; 29+ messages in thread
From: Benjamin Herrenschmidt @ 2018-07-27  8:18 UTC (permalink / raw)
  To: Cédric Le Goater, David Gibson
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On Fri, 2018-07-27 at 09:16 +0200, Cédric Le Goater wrote:
> > I'd have to remember how PQ works on P8 ... my gut feeling is that we
> > should resend if P=1 but I'm no 100% certain.
> 
> This is not exactly what the code does. To force a resend, it ignores 
> P but if Q=1, it bails out without doing anything, like a normal trigger 
> would do. So I think that in the resend case we should ignore Q as well.

Not too sure. We might need to swallow the resend if PQ became 01.

I don't know for sure what the HW does. Does it unconditionally resend
or does it re-evaluate the priority etc... it's unclear to me.

But yes, we should still resend if PQ=11

Cheers,
Ben.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-27  8:08         ` Benjamin Herrenschmidt
@ 2018-07-27  8:25           ` Cédric Le Goater
  2018-07-27  8:43             ` Benjamin Herrenschmidt
  0 siblings, 1 reply; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-27  8:25 UTC (permalink / raw)
  To: Benjamin Herrenschmidt, David Gibson
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On 07/27/2018 10:08 AM, Benjamin Herrenschmidt wrote:
> On Fri, 2018-07-27 at 15:32 +1000, David Gibson wrote:
>>
>>>> What is this pci bridge representing?  I know PCI-e PHBs typically
>>>> have a pseudo P2P bridge right under them, but isn't that represnted
>>>> by the root complex above?
>>>
>>> This is the legacy pci bridge under the pcie bus.
>>
>> Ah, ok.  Didn't realise there was a vanilla PCI bridge built in.
> 
> There isn't. That should probably be created by the machine.

Each PHB creates a pci-bridge device and the PCI bus that comes with it. 
It makes things easier to define PCI devices. 

It is still quite complex ... Here is a sample :

qemu-system-ppc64 -m 2G -machine powernv \
  -cpu POWER8 -smp 2,cores=2,threads=1 -accel tcg,thread=multi \
  -kernel ./zImage.epapr -initrd ./rootfs.cpio.xz -bios ./skiboot.lid \
  \
  -device megasas,id=scsi0,bus=pci.0,addr=0x1 \
  -drive file=./rhel7-ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none \
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2 \
  \
  -device ich9-ahci,id=sata0,bus=pci.1,addr=0x1 \
  -drive file=./ubuntu-ppc64le.qcow2,if=none,id=drive0,format=qcow2,cache=none \
  -device ide-hd,bus=sata0.0,unit=0,drive=drive0,id=ide,bootindex=1 \
  -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.1,addr=0x2 \
  -netdev bridge,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=net0 \
  -device nec-usb-xhci,bus=pci.1,addr=0x7 \


 


>>> Here is is the qdev hierarchy :
>>>
>>>   dev: pnv-phb3, id ""
>>>     index = 0 (0x0)
>>>     chip-id = 0 (0x0)
>>>     bus: phb3-root-bus
>>>       type pnv-phb3-root-bus
>>>       dev: pnv-phb3-rc, id ""
>>>         power_controller_present = true
>>>         chassis = 0 (0x0)
>>>         slot = 1 (0x1)
>>>         port = 0 (0x0)
>>>         aer_log_max = 8 (0x8)
>>>         addr = 00.0
>>>         romfile = ""
>>>         rombar = 1 (0x1)
>>>         multifunction = false
>>>         command_serr_enable = true
>>>         x-pcie-lnksta-dllla = true
>>>         x-pcie-extcap-init = true
>>>         class PCI bridge, addr 00:00.0, pci id 1014:03dc (sub 0000:0000)
>>>         bus: pcie.0
>>>           type PCIE
>>>           dev: pci-bridge, id ""
>>>             chassis_nr = 128 (0x80)
>>>             msi = "off"
>>>             shpc = false
>>>             addr = 00.0
>>>             romfile = ""
>>>             rombar = 1 (0x1)
>>>             multifunction = false
>>>             command_serr_enable = true
>>>             x-pcie-lnksta-dllla = true
>>>             x-pcie-extcap-init = true
>>>             class PCI bridge, addr 00:00.0, pci id 1b36:0001 (sub 0000:0000)
>>>             bus: pci.0
>>>               type PCI
>>>
>>> [ ... ]
>>>
>>>>> +static const PropertyInfo pnv_phb3_phb_id_propinfo = {
>>>>> +    .name = "irq",
>>>>> +    .get = pnv_phb3_get_phb_id,
>>>>> +    .set = pnv_phb3_set_phb_id,
>>>>> +};
>>>>
>>>> Can't you use a static DeviceProps style property for this, which is a
>>>> bit simpler?
>>>
>>> OK. We will address user creatable PHBs in some other way. Most
>>> certainly in the realize routine like you suggested.
>>>
>>> Thanks,
>>>
>>> C. 
>>>
>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-27  8:25           ` Cédric Le Goater
@ 2018-07-27  8:43             ` Benjamin Herrenschmidt
  2018-07-27  9:19               ` Cédric Le Goater
  0 siblings, 1 reply; 29+ messages in thread
From: Benjamin Herrenschmidt @ 2018-07-27  8:43 UTC (permalink / raw)
  To: Cédric Le Goater, David Gibson
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On Fri, 2018-07-27 at 10:25 +0200, Cédric Le Goater wrote:
> Each PHB creates a pci-bridge device and the PCI bus that comes with it. 
> It makes things easier to define PCI devices. 
> 
> It is still quite complex ... Here is a sample :
> 
> qemu-system-ppc64 -m 2G -machine powernv \
>   -cpu POWER8 -smp 2,cores=2,threads=1 -accel tcg,thread=multi \
>   -kernel ./zImage.epapr -initrd ./rootfs.cpio.xz -bios ./skiboot.lid \
>   \
>   -device megasas,id=scsi0,bus=pci.0,addr=0x1 \
>   -drive file=./rhel7-ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none \
>   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2 \
>   \
>   -device ich9-ahci,id=sata0,bus=pci.1,addr=0x1 \
>   -drive file=./ubuntu-ppc64le.qcow2,if=none,id=drive0,format=qcow2,cache=none \
>   -device ide-hd,bus=sata0.0,unit=0,drive=drive0,id=ide,bootindex=1 \
>   -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.1,addr=0x2 \
>   -netdev bridge,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=net0 \
>   -device nec-usb-xhci,bus=pci.1,addr=0x7 \

I don't understand why. That means you can't put emulated (or real)
PCIe device below it ? Why force them down the path of having a bridge
to legacy PCI always ?

My original intent was to have one such bridge in a machine for the
various default PCI devices qemu has that aren't (yet) PCIe (and also
for testing :-) but I never thought we'd throw one onto every PCIe bus.

Ben.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-27  8:43             ` Benjamin Herrenschmidt
@ 2018-07-27  9:19               ` Cédric Le Goater
  2018-07-30  8:56                 ` Cédric Le Goater
  0 siblings, 1 reply; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-27  9:19 UTC (permalink / raw)
  To: Benjamin Herrenschmidt, David Gibson
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On 07/27/2018 10:43 AM, Benjamin Herrenschmidt wrote:
> On Fri, 2018-07-27 at 10:25 +0200, Cédric Le Goater wrote:
>> Each PHB creates a pci-bridge device and the PCI bus that comes with it. 
>> It makes things easier to define PCI devices. 
>>
>> It is still quite complex ... Here is a sample :
>>
>> qemu-system-ppc64 -m 2G -machine powernv \
>>   -cpu POWER8 -smp 2,cores=2,threads=1 -accel tcg,thread=multi \
>>   -kernel ./zImage.epapr -initrd ./rootfs.cpio.xz -bios ./skiboot.lid \
>>   \
>>   -device megasas,id=scsi0,bus=pci.0,addr=0x1 \
>>   -drive file=./rhel7-ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none \
>>   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2 \
>>   \
>>   -device ich9-ahci,id=sata0,bus=pci.1,addr=0x1 \
>>   -drive file=./ubuntu-ppc64le.qcow2,if=none,id=drive0,format=qcow2,cache=none \
>>   -device ide-hd,bus=sata0.0,unit=0,drive=drive0,id=ide,bootindex=1 \
>>   -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.1,addr=0x2 \
>>   -netdev bridge,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=net0 \
>>   -device nec-usb-xhci,bus=pci.1,addr=0x7 \
> 
> I don't understand why. That means you can't put emulated (or real)
> PCIe device below it ?

Well, skiboot does seem to find them. But that's not a good reason.
I will dig in.

> Why force them down the path of having a bridge to legacy PCI always ?
>
> My original intent was to have one such bridge in a machine for the
> various default PCI devices qemu has that aren't (yet) PCIe (and also
> for testing :-) but I never thought we'd throw one onto every PCIe bus.

OK. Let's get of rid it and let's find why the FW doesn't see the PCIe
devices.

Thanks,

C.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [Qemu-devel] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge
  2018-07-27  9:19               ` Cédric Le Goater
@ 2018-07-30  8:56                 ` Cédric Le Goater
  0 siblings, 0 replies; 29+ messages in thread
From: Cédric Le Goater @ 2018-07-30  8:56 UTC (permalink / raw)
  To: Benjamin Herrenschmidt, David Gibson
  Cc: qemu-ppc, qemu-devel, Marcel Apfelbaum, Andrea Bolognani,
	Michael S. Tsirkin

On 07/27/2018 11:19 AM, Cédric Le Goater wrote:
> On 07/27/2018 10:43 AM, Benjamin Herrenschmidt wrote:
>> On Fri, 2018-07-27 at 10:25 +0200, Cédric Le Goater wrote:
>>> Each PHB creates a pci-bridge device and the PCI bus that comes with it. 
>>> It makes things easier to define PCI devices. 
>>>
>>> It is still quite complex ... Here is a sample :
>>>
>>> qemu-system-ppc64 -m 2G -machine powernv \
>>>   -cpu POWER8 -smp 2,cores=2,threads=1 -accel tcg,thread=multi \
>>>   -kernel ./zImage.epapr -initrd ./rootfs.cpio.xz -bios ./skiboot.lid \
>>>   \
>>>   -device megasas,id=scsi0,bus=pci.0,addr=0x1 \
>>>   -drive file=./rhel7-ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none \
>>>   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2 \
>>>   \
>>>   -device ich9-ahci,id=sata0,bus=pci.1,addr=0x1 \
>>>   -drive file=./ubuntu-ppc64le.qcow2,if=none,id=drive0,format=qcow2,cache=none \
>>>   -device ide-hd,bus=sata0.0,unit=0,drive=drive0,id=ide,bootindex=1 \
>>>   -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.1,addr=0x2 \
>>>   -netdev bridge,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=net0 \
>>>   -device nec-usb-xhci,bus=pci.1,addr=0x7 \
>>
>> I don't understand why. That means you can't put emulated (or real)
>> PCIe device below it ?
> 
> Well, skiboot does seem to find them. But that's not a good reason.
> I will dig in.

Nothing is wrong in skiboot, that was my misunderstanding of the overall 
device layout.

Each PHB3 device exposes a single PCIE root port on which a *single* device 
can be plugged. It can be a any adapter or a bridge/switch. The QEMU device 
hierarchy is rather deep if you want to have multiple devices on a single 
PHB3 :

dev: pnv-phb3
  bus: phb3-root-bus
    dev: phb3-root-port
      bus: pcie 
         dev: pcie-bridge (only one device allowed) 
            bus: pcie 
               dev: e1000e
               dev: ...


So it makes the command line rather heavy :/ or you need to have 
more phb3s to define an adapter on each. How about 3 per chip ?


Here is a quick survey,

On OpenPower systems: 

  palmetto:  3 PHBs - one 16x, one 8x, one 4x backed by a switch
             with the BMC devices 

  habanero:  3 PHBs - one 16x, one 8x, one 8x backed by a switch 
             with the BMC devices 

  firestone: socket 0 : 2 16x PHBs
             socket 1 : 3 PHBs - one 16x, one 8x, one 8x backed  
                        by a switch with the BMC devices  

  garrison:  socket 0 : 1 16x PHB, 2 8x PHB linked to the GPUs, 1 8x PHB, 
             socket 1 : 1 16x PHB, 2 8x PHB linked to the GPUs, 1 8x PHB
	                backed by a switch with the BMC devices  


On IBM systems :

  tuletas:   2 16x PHB, 2 8x PHB, backed with a switch. 
  

Thanks,

C. 

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2018-07-30  8:56 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-28  8:36 [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge Cédric Le Goater
2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 1/2] " Cédric Le Goater
2018-07-09  7:22   ` Cédric Le Goater
2018-07-18  6:13     ` David Gibson
2018-07-18  6:12   ` David Gibson
2018-07-18  8:03     ` Benjamin Herrenschmidt
2018-07-23  4:16       ` David Gibson
2018-07-23 23:55         ` Benjamin Herrenschmidt
2018-07-24  2:14           ` David Gibson
2018-07-24  3:49             ` Benjamin Herrenschmidt
2018-07-24  4:10               ` David Gibson
2018-07-23 21:35     ` Cédric Le Goater
2018-07-23 21:37     ` Cédric Le Goater
2018-07-24  1:29       ` David Gibson
2018-07-23 21:38     ` Cédric Le Goater
2018-07-24 12:18     ` Cédric Le Goater
2018-07-27  5:32       ` David Gibson
2018-07-27  7:33         ` Cédric Le Goater
2018-07-27  8:08         ` Benjamin Herrenschmidt
2018-07-27  8:25           ` Cédric Le Goater
2018-07-27  8:43             ` Benjamin Herrenschmidt
2018-07-27  9:19               ` Cédric Le Goater
2018-07-30  8:56                 ` Cédric Le Goater
2018-07-26  9:03   ` Cédric Le Goater
2018-07-26 22:36     ` Benjamin Herrenschmidt
2018-07-27  7:16       ` Cédric Le Goater
2018-07-27  8:18         ` Benjamin Herrenschmidt
2018-06-28  8:36 ` [Qemu-devel] [PATCH v2 2/2] ppc/pnv: make the PHB3 devices user creatable Cédric Le Goater
2018-07-01 18:33 ` [Qemu-devel] [PATCH v2 0/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge no-reply

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.