All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] PCIe Transaction handling in Qemu
@ 2010-12-21 20:24 Adnan Khaleel
  2010-12-22 10:40 ` [Qemu-devel] " Isaku Yamahata
  2010-12-22 11:24 ` [Qemu-devel] " Paul Brook
  0 siblings, 2 replies; 4+ messages in thread
From: Adnan Khaleel @ 2010-12-21 20:24 UTC (permalink / raw)
  To: qemu-devel; +Cc: yamahata

[-- Attachment #1: Type: text/plain, Size: 3276 bytes --]

Hello, 


I have a question regarding how Qemu PCIe devices handle Config Transactions vs Memory Transactions (assuming the PCI device is setup to act as PCI_BASE_ADDRESS_SPACE_MEMORY).


I'm using portions of hw/cirrus_vga.c to make my point,



static PCIDeviceInfo cirrus_vga_info = {
    .qdev.name    = "cirrus-vga",
    .qdev.desc    = "Cirrus CLGD 54xx VGA",
    .qdev.size    = sizeof(PCICirrusVGAState),
    .qdev.vmsd    = &vmstate_pci_cirrus_vga,
    .init         = pci_cirrus_vga_initfn,
    .romfile      = VGABIOS_CIRRUS_FILENAME,
    .config_write = pci_cirrus_write_config,
};


PCIDeviceInfo allows for custom .config_write (& config_read) handler as shown above. Any pci config operations operations initiated via legacy I/O operations will use these config handlers.


The MMIO regions and handlers are mapped as shown below:



static uint32_t cirrus_vga_mem_readb(void *opaque, target_phys_addr_t addr)
{
:
} and so on for the other mmio handlers




static CPUReadMemoryFunc * const cirrus_vga_mem_read[3] = {
    cirrus_vga_mem_readb,
    cirrus_vga_mem_readw,
    cirrus_vga_mem_readl,
};


static CPUWriteMemoryFunc * const cirrus_vga_mem_write[3] = {
    cirrus_vga_mem_writeb,
    cirrus_vga_mem_writew,
    cirrus_vga_mem_writel,
};


static void cirrus_init_common(CirrusVGAState * s, int device_id, int is_pci)

{
:
:

    s->vga.vga_io_memory = cpu_register_io_memory(cirrus_vga_mem_read,
                                                  cirrus_vga_mem_write, s);
:
}



static void cirrus_pci_mmio_map(PCIDevice *d, int region_num,
    pcibus_t addr, pcibus_t size, int type)
{
    CirrusVGAState *s = &DO_UPCAST(PCICirrusVGAState, dev, d)->cirrus_vga;


    cpu_register_physical_memory(addr, CIRRUS_PNPMMIO_SIZE,
     s->cirrus_mmio_io_addr);
}



static int pci_cirrus_vga_initfn(PCIDevice *dev)
{
:
:
     cirrus_init_common(..)
:
     if (device_id == CIRRUS_ID_CLGD5446) {
         pci_register_bar((PCIDevice *)d, 1, CIRRUS_PNPMMIO_SIZE,
                          PCI_BASE_ADDRESS_SPACE_MEMORY, cirrus_pci_mmio_map);
     }
     return 0;
}


I have some questions about PCIe operations sssuming the device has MMIO handlers involved (as shown above).
1. Will all PCIe config operations ALWAYS use the installed config handlers? Or can PCIe config operations use the MMIO handlers?
2. Assuming that both PCI config and MMIO operations can use the MMIO handlers, is there any way I can identify if a transaction is a config or a memory transaction?
3.a. What address is passed on the MMIO handlers for config and MMIO operations? From pci_data_write in pci_host.c, it appears that config operations send only the offset into the config region. I couldn't determine what address is passed for MMIO operations.
   b. Is it an offset from the BAR for MMIO operations?
   c. How do I get the full physical address?
   d. What address does a PCIe device expect to see - physical or offset for?
   e. Is there anyway I can find out what the bus and device numbers are once inside the config and MMIO handlers? i.e once the execution has reached the pci_cirrus_write_config() or cirrus_vga_mem_readb(..) from the code above?


Thanks


Adnan

[-- Attachment #2: Type: text/html, Size: 7964 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [Qemu-devel] Re: PCIe Transaction handling in Qemu
  2010-12-21 20:24 [Qemu-devel] PCIe Transaction handling in Qemu Adnan Khaleel
@ 2010-12-22 10:40 ` Isaku Yamahata
  2010-12-22 11:24 ` [Qemu-devel] " Paul Brook
  1 sibling, 0 replies; 4+ messages in thread
From: Isaku Yamahata @ 2010-12-22 10:40 UTC (permalink / raw)
  To: Adnan Khaleel; +Cc: qemu-devel

On Tue, Dec 21, 2010 at 02:24:29PM -0600, Adnan Khaleel wrote:
> Hello, 

Hi.


> I have a question regarding how Qemu PCIe devices handle Config Transactions vs
> Memory Transactions (assuming the PCI device is setup to act
> as PCI_BASE_ADDRESS_SPACE_MEMORY).
> 
> I'm using portions of hw/cirrus_vga.c to make my point,

If you can send out what you have instead of mimicked
example, it would help to figure out what you are trying to do.


> I have some questions about PCIe operations sssuming the device has MMIO
> handlers involved (as shown above).
> 1. Will all PCIe config operations ALWAYS use the installed config handlers? Or
> can PCIe config operations use the MMIO handlers?

MMIO on MMCONFIG area are routed to write/read config handler.
On the other hand MMIO on memory BAR is routed to mmio hanlder you pictured.
NOTE: the upstream qemu lacks q35 chipset support, so guest can NOT do
      MMIO on MMCONFIG area.

> 2. Assuming that both PCI config and MMIO operations can use the MMIO handlers,
> is there any way I can identify if a transaction is a config or a memory
> transaction?
> 3.a. What address is passed on the MMIO handlers for config and MMIO
> operations? From pci_data_write in pci_host.c, it appears that config
> operations send only the offset into the config region. I couldn't determine
> what address is passed for MMIO operations.
>    b. Is it an offset from the BAR for MMIO operations?
>    c. How do I get the full physical address?
>    d. What address does a PCIe device expect to see - physical or offset for?
>    e. Is there anyway I can find out what the bus and device numbers are once
> inside the config and MMIO handlers? i.e once the execution has reached
> the pci_cirrus_write_config() or cirrus_vga_mem_readb(..) from the code above?

offset in configuration space of each pcie function is passed to
write/read config handler
physical address is passed to mmio handler of memory BAR.
-- 
yamahata

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [Qemu-devel] PCIe Transaction handling in Qemu
  2010-12-21 20:24 [Qemu-devel] PCIe Transaction handling in Qemu Adnan Khaleel
  2010-12-22 10:40 ` [Qemu-devel] " Isaku Yamahata
@ 2010-12-22 11:24 ` Paul Brook
  1 sibling, 0 replies; 4+ messages in thread
From: Paul Brook @ 2010-12-22 11:24 UTC (permalink / raw)
  To: qemu-devel, adnan; +Cc: yamahata

> I have some questions about PCIe operations sssuming the device has MMIO
> handlers involved (as shown above).

> 1. Will all PCIe config operations
> ALWAYS use the installed config handlers? Or can PCIe config operations
> use the MMIO handlers? 

Access to PCI config space is provided by the PCI host bridge. It has nothing 
to do with any memory BARs the device may have.  The host bridge may expose 
this in any way it chooses, including but not limited to ISA IO ports or a 
memory mapped region of its own.  Ether way the device doesn't care.

> 2. Assuming that both PCI config and MMIO
> operations can use the MMIO handlers, is there any way I can identify if a
> transaction is a config or a memory transaction?

Incorrect assumption. Memory and Config accesses ae completely separate.

> 3.a. What address is
> passed on the MMIO handlers for config and MMIO operations? From
> pci_data_write in pci_host.c, it appears that config operations send only
> the offset into the config region. I couldn't determine what address is
> passed for MMIO operations. b. Is it an offset from the BAR for MMIO
> operations?

Th offset from the start of the region.

>    c. How do I get the full physical address?

You don't.  "Full physical address" is a fairly ill defined term. Physical 
addresses are local to a particular bus. It's common for CPU/ram and each PCI 
bus to have completely independent physical address spaces, with the host 
bridge providing mapping between the two.

>    d. What address does a PCIe device expect to see - physical or offset
> for? 

Offset. Old versions of qemu used to pass the cpu physical address. This was a 
bug.

> e. Is there anyway I can find out what the bus and device numbers are
> once inside the config and MMIO handlers? i.e once the execution has
> reached the pci_cirrus_write_config() or cirrus_vga_mem_readb(..) from the
> code above?

No. The device does not, and should not know this. 

Paul

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [Qemu-devel] Re: PCIe Transaction handling in Qemu
@ 2010-12-27 18:12 Adnan Khaleel
  0 siblings, 0 replies; 4+ messages in thread
From: Adnan Khaleel @ 2010-12-27 18:12 UTC (permalink / raw)
  To: Isaku Yamahata, Paul Brook; +Cc: qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 8354 bytes --]

 
Yamahata and Paul, thank you very much for your responses. I have some followup questions below.


Thanks for all your assistance!


Adnan > I have a question regarding how Qemu PCIe devices handle Config Transactions vs
  > Memory Transactions (assuming the PCI device is setup to act
  > as PCI_BASE_ADDRESS_SPACE_MEMORY).
  > 
  > I'm using portions of hw/cirrus_vga.c to make my point,
  
  If you can send out what you have instead of mimicked
  example, it would help to figure out what you are trying to do.
  
I was trying to keep things simple with common code that was well established inside Qemu. I've attached the code from my PCIe device to help you better understand what I'm trying to do.
I'm using Qemu to drive a PCIe device model. Here is a diagrammatic representation
______      ___________________         ___________________
Qemu  |<-->|PCIe Model Wrapper |<----->| PCIe Device Model |
______|    |___________________|       |___________________|


The PCIe Device model is a complete GPU model that communicates via a simulated PCIe interface i.e. the model represents a PCIe card that would plug into a PCIe slot and expects to see PCIe transactions to drive it. In other words, the PCIe Device model thinks its connected to a PCI-Bridge so I have to mimic this partially with the PCIe Model wrapper. It is important that you understand this since its very central to the issues I'm trying to better comprehend. I don't have the source code to the device model, hence my many challenges.


I've written the PCIe model wrapper (the attached file), this wrapper acts as a proxy for the Device Model and is what Qemu "sees" as an PCI endpoint. The PCIe Model Wrapper is modelled as a Qemu PCIe device and includes PCI configuration space registers, however the TRUE PCI config registers are contained with the PCIe Device Model. 


So in a nutshell this is what I'm trying to do: 
- When the Guest OS inside Qemu does a PCIe config read/write, the PCIe model wrapper has to intercept that call, construct a valid PCIe config transaction and send it to the PCIe device model. 
- Similarly for a PCIe MMIO transaction i.e, the wrapper has to construct a PCIe transaction and send it to the device model.


I hope this puts into perspective why I was asking the questions about how I can definitively identify if a request is a config request or if its a MMIO request because I need to synthesize the appropriate PCIe transactions to send to the Model.

> I have some questions about PCIe operations sssuming the device has MMIO
  > handlers involved (as shown above).
  > 1. Will all PCIe config operations ALWAYS use the installed config handlers? Or
  > can PCIe config operations use the MMIO handlers?
  
  MMIO on MMCONFIG area are routed to write/read config handler.
  On the other hand MMIO on memory BAR is routed to mmio hanlder you pictured.
  NOTE: the upstream qemu lacks q35 chipset support, so guest can NOT do
        MMIO on MMCONFIG area.
  
I am using the q35 chipset that you (Yamahata) have been working on. So does that mean that guests OS with the qemu with q35 can only access the legacy PCI config space (256 bytes) and not the full PCIe 4k config space? The reason I ask is because of an AMD document I found


http://developer.amd.com/Assets/pci%20-%20pci%20express%20configuration%20space%20access.pdf


Here is the relevant text:


3.3 MMIO access with different Linux Kernel versions Support for MMIO access for PCI configuration space depends on the Linux Kernel version and configuration, and the existence of an MCFG ACPI table. IO access to PCI configuration space is always possible and will be used if MMIO access is not available. To access extended configuration space (between byte 256 and 4095) MMIO access is required.
:
Note: In both cases Linux supports the IO access method as a fallback if MMIO access is not possible (e.g. if MCFG ACPI table is missing). 


A kernel (with version prior to and including 2.6.25) with enabled MMIO access method is able to do MMIO access if and only if the BIOS provides a valid MCFG ACPI table and if the configuration space is "E820-reserved". If these requirements are not met, IO access is used and the kernel logs an error message like this PCI: BIOS Bug: MCFG area at e0000000 is not E820-reserved PCI: Not using MMCONFIG. PCI: Using configuration type 1 Beginning with kernel version 2.6.25 Linux always uses the IO access method to access configuration space below byte 256. Thus MMIO access is used only for extended configuration space.


  > 2. Assuming that both PCI config and MMIO operations can use the MMIO handlers,
  > is there any way I can identify if a transaction is a config or a memory
  > transaction?
Here is a code snippet where I register the config handlers:



static PCIDeviceInfo pcie_msix_info = {
    .qdev.name      = PCIE_MSIX_DEVICE,
    .qdev.desc      = "PCIE MSIX device template",
    .qdev.size      = sizeof(PCIE_MSIX_DEVState),
    .qdev.reset     = pcie_msix_reset,
    .qdev.vmsd      = &vmstate_pcie_msix,
    .is_express     = 1,
    .config_read    = pcie_msix_read_config,
    .config_write   = pcie_msix_write_config,
    .init           = pcie_msix_initfn,
    .exit           = pcie_msix_exitfn,
    .qdev.props     = (Property[]) {
        DEFINE_PROP_UINT32("vectors", PCIE_MSIX_DEVState, vectors, PCIE_MSIX_MSI_NR_VECTOR),        
        DEFINE_PROP_UINT16("aer_log_max", PCIESlot,
                            port.br.dev.aer_log.log_max,
                            PCIE_AER_LOG_MAX_DEFAULT),
        DEFINE_PROP_END_OF_LIST(),
    }
};


Here is a code snippet where I register the mmio handlers:



static CPUWriteMemoryFunc * const pcie_msix_mem_write_fn[] = {
    pcie_msix_mem_writeb, pcie_msix_mem_writew, pcie_msix_mem_writel, pcie_msix_mem_writed
};


static CPUReadMemoryFunc * const pcie_msix_mem_read_fn[] = {
    pcie_msix_mem_readb, pcie_msix_mem_readw, pcie_msix_mem_readl, pcie_msix_mem_readd
};


static void 
pcie_msix_mem_map(PCIDevice *dev, int region_num, 
                  pcibus_t addr, pcibus_t size, int type)
{
    uint32_t msix_mem_bar = 0;
    PCIE_MSIX_DEVState *d = DO_UPCAST(PCIE_MSIX_DEVState, dev, dev);
    cpu_register_physical_memory(addr, BAR_Regions[msix_mem_bar][0], d->mmio_index);
}


Which I register via the following call in the pcie_msix_initfn(PCIDevice *pci_dev) function


pci_register_bar(&d->dev, msix_mem_bar, BAR_Regions[msix_mem_bar][0], BAR_Regions[msix_mem_bar][1], pcie_msix_mem_map);


So are my following assumption true then for my PCIe device i.e the wrapper?
All config requests WILL be handled by the config handlers and,
All MMIO requests will be handled by the mmio handlers (given that the q35 model does not support MMIO on MMCONFIG)
  
  > 3.a. What address is passed on the MMIO handlers for config and MMIO
  > operations? From pci_data_write in pci_host.c, it appears that config
  > operations send only the offset into the config region. I couldn't determine
  > what address is passed for MMIO operations.
  >    b. Is it an offset from the BAR for MMIO operations?
  >    c. How do I get the full physical address?
  >    d. What address does a PCIe device expect to see - physical or offset for?
  >    e. Is there anyway I can find out what the bus and device numbers are once
  > inside the config and MMIO handlers? i.e once the execution has reached
  > the pci_cirrus_write_config() or cirrus_vga_mem_readb(..) from the code above?
  
  offset in configuration space of each pcie function is passed to
  write/read config handler
  physical address is passed to mmio handler of memory BAR.
When I examine the address on calls to the PCIe Device Wrapper config handlers, they start at 0x0 so they clearly are offsets. However, I also see addresses from 0x0 in the MMIO handlers. Paul's response indicated that they were offsets so I'm inclined to believe that the address is an offset from the start of the BAR. So let me ask my question again - when I'm synthesizing the PCIe transaction to send to the PCIe device model, do I have to give it a full address i.e. bus,dev,fn for config transactions and full physical address for MMIO transactions?


  -- 
  yamahata
      

[-- Attachment #1.2: Type: text/html, Size: 15526 bytes --]

[-- Attachment #2: pcie_msix_template_1.c --]
[-- Type: text/plain, Size: 20392 bytes --]

/*
 * pcie_template_device.c
 *
 * Copyright (c) 2010 Adnan Khaleel <akhaleel at cray com>
 *                    Cray Inc
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License along
 * with this program; if not, see <http://www.gnu.org/licenses/>.
 */

/* This is essentially an example for a device that utilises PCIe and MSI-X */

#include "hw.h"
#include "pc.h"
#include "pci_ids.h"
#include "msi.h"
#include "msix.h"
#include "pcie.h"

#include "pcie_msix_template.h"
#include "pcie_pkt.h"

extern int             pcie_server_fd;
extern struct hostent *pcie_server;
extern uint64          pcie_transId;
extern bool            pcie_server_connected;

#define BYTE  1
#define WORD  2*BYTE
#define LWORD 4*BYTE
#define DWORD 8*BYTE


/* Debug code */
#define DEBUG_PCIE_MSIX_TEMPLATE

#ifdef DEBUG_PCIE_MSIX_TEMPLATE
#define SPLIT64(x) (uint32_t)((x >> 32) & 0xffffffff), (uint32_t)(x & 0xffffffff)
#define PRINT_DEBUG(format, ...) printf(format, __VA_ARGS__);
#define PRINT_DEBUG0(format) printf(format);
#else
#define SPLIT64(x)
#define PRINT_DEBUG(format, ...)
#define PRINT_DEBUG0(format)
#endif

#define DEFINE_DEVICE_PROPERTIES(_state, _conf)

#define PCIE_MSIX_VID                   PCI_VENDOR_ID_CRAY
#define PCIE_MSIX_DID                   PCI_DEVICE_ID_CRAY_ASIC
#define PCIE_MSIX_SS_DID                0x100
#define PCIE_MSIX_VERSION               0x1
#define PCIE_MSIX_MSI_SUPPORTED_FLAGS   PCI_MSI_FLAGS_64BIT
#define PCIE_MSIX_MSI_NR_VECTOR         1        
#define PCIE_MSIX_MSI_OFFSET            0x70
#define PCIE_MSIX_SSVID_OFFSET          0x80
#define PCIE_MSIX_SVID                  0
#define PCIE_MSIX_SSID                  0
#define PCIE_MSIX_EXP_OFFSET            0x90
#define PCIE_MSIX_AER_OFFSET            0x100

/* PCI BUS commands */
#define CMD_IOREAD	        0x02
#define CMD_IOWRITE	        0x03
#define CMD_MEMREAD	        0x06
#define CMD_MEMWRITE	    0x07
#define CMD_CONFIGREAD	    0x0A
#define CMD_CONFIGWRITE	    0x0B

// PCI device header
uint32_t g_cfg_init[0x100] =
{
    //            index : offset
    0x030117db, // 0x00 : 0x00: Vendor ID and device ID
    0x00100007, // 0x01 : 0x04: Command and status
    0x00000010, // 0x02 : 0x08: Revision ID and class code
    0x00800008, // 0x03 : 0x0c: BIST, header type, latency time and cache size
    0x0000000c, // 0x04 : 0x10: BAR 0
    0x00000020, // 0x05 : 0x14: BAR 1
    0x00000000, // 0x06 : 0x18: BAR 2
    0x00000000, // 0x07 : 0x1c: BAR 3
    0x00000000, // 0x08 : 0x20: BAR 4
    0x00000000, // 0x09 : 0x24: BAR 5
    0x00000000, // 0x0a : 0x28
    0x000017db, // 0x0b : 0x2c: Subsystem vendor ID and subsystem ID
    0x00000000, // 0x0c : 0x30: Expansion ROM base address
    0x00000080, // 0x0d : 0x34: Capabilities pointer
    0x00000000, // 0x0e : 0x38
    0x0000010b, // 0x0f : 0x3c: Interrupt line and interrupt pin
    0, 0, 0, 0, // 0x10 : 0x40
    0, 0, 0, 0, // 0x14 : 0x50
    0, 0, 0, 0, // 0x18 : 0x60
    0, 0, 0, 0, // 0x1c : 0x70
    0x0003b001, // 0x20 : 0x80: Power management capabilities
    0x00000008, // 0x21 : 0x84: Power management control/status
    0x00000000, // 0x22 : 0x88
    0x00000000, // 0x23 : 0x8c
    0, 0, 0, 0, // 0x24 : 0x90
    0, 0, 0, 0, // 0x28 : 0xa0
    0x003fc011, // 0x2c : 0xb0: MSI-X control
    0x00402000, // 0x2d : 0xb4: MSI-X table offset
    0x00403000, // 0x2e : 0xb8: MSI-X pending interrupt
    0x00000000, // 0x2f : 0xbc
    0x00010010, // 0x30 : 0xc0: PCIe capability list register
    0x00008120, // 0x31 : 0xc4: PCIe device capabilities
    0x00092100, // 0x32 : 0xc8: PCIe device control and status
    0x00004411, // 0x33 : 0xcc: Link capabilities
    0x00110000, // 0x34 : 0xd0: Link control and status
    0x00000000, // 0x35 : 0xd4
    0x00000000, // 0x36 : 0xd8
    0x00000000, // 0x37 : 0xdc
    0x00000000, // 0x38 : 0xe0
    0x00000012, // 0x39 : 0xe4: PCIe device capabilities 2
    0x00000000, // 0x3a : 0xe8: PCIe device control and status 2
    0x00000000, // 0x3b : 0xec: Link capabilities 2
    0x00000000, // 0x3c : 0xf0: Link control and status 2
};

static const unsigned long long BAR_Regions[6][2] = 
{
    // len , type 0x2000000000ull
    { 0x2000000000ull, PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64 | PCI_BASE_ADDRESS_MEM_PREFETCH} ,  //BAR0,      
    { 0, 0} , // BAR1
    { 0, 0} , // BAR2,
    { 0, 0} , // BAR3
    { 0, 0} , // BAR4
    { 0, 0} , // BAR5    
};

#define TIMEOUT_SEC  0
#define TIMEOUT_USEC 50

extern int do_remote_pcie_model_socket_read(void* pcie_trans, int size, int to_sec, int to_usec, bool startup, int64_t loop_count);
extern int do_remote_pcie_model_socket_write(void* pcie_trans, int size, int to_sec, int to_usec, bool startup, int64_t loop_count);

typedef struct PCIE_MSIX_DEVState_St {
    PCIDevice dev;
    int mmio_index;

    uint32_t intrmask;
    uint32_t intrstatus;
    uint32_t doorbell;

    uint32_t vectors;
    uint32_t features;

} PCIE_MSIX_DEVState;

static uint64_t 
do_socket_transaction(void *opaque, bool rd, uint64_t addr, uint64_t val, int len)
{
    int bar_region = 0, i, err;
    uint32_t reg_base, reg_size, base_addr;
    PCIDevice *d = opaque;
    PCIe_cmd_t pcie_trans;

    reg_base  = d->dev.io_regions[bar_region].addr;
    reg_size  = d->dev.io_regions[bar_region].size;
    base_addr = d->dev.io_regions[bar_region].addr;

    pcie_trans.pktType     = (rd) ? CFG_RD_0 : CFG_WR_0;
    pcie_trans.addr64      = addr;
    pcie_trans.transId     = pcie_transId++;
    for (i = 0; i < len; i++)
        pcie_trans.data[i] = (rd) ? 0 : ((val >> (i*8)) & 0xff);
    pcie_trans.data_len    = len;
    pcie_trans.stop_server = false;
    pcie_trans.start_delta = 0;
    pcie_trans.isAlive     = false;
    pcie_trans.valid       = true;
    pcie_trans.aries_req   = false;

    if (rd) {
        PRINT_DEBUG("%s: Cfg_Rd Addr=0x%08x%08x ReqId=%lld bar=%d\n", __FUNCTION__, SPLIT64(addr64), pcie_trans.transId, bar_region);
    }
    else {
        PRINT_DEBUG("%s: Cfg_Wr Addr=0x%08x%08x ReqId=%lld bar=%d Data=0x", __FUNCTION__, SPLIT64(addr64), pcie_trans.transId, bar_region);
        for (i = 0; i < len; i++)
            PRINT_DEBUG("%02x", pcie_trans.data[i]);
        PRINT_DEBUG0("\n");
    }

    if (pcie_server_connected) {
        err = do_remote_pcie_model_socket_write((void*)&pcie_trans, sizeof(PCIe_cmd_t), TIMEOUT_SEC, TIMEOUT_USEC, false, 0);
        if (err > 0) {
            if (rd) {
                PCIe_cmd_t cmpltd_cmd;
                cmpltd_cmd.valid = false;
                err = do_remote_pcie_model_socket_read((void*)&cmpltd_cmd, sizeof(PCIe_cmd_t), TIMEOUT_SEC, TIMEOUT_USEC, false, 0);
                if ((err > 0) && (cmpltd_cmd.valid)) {
                    if ((cmpltd_cmd.pktType == CMPLT_D) && 
                        (pcie_trans.transId == cmpltd_cmd.transId)) 
                    {
                        val = cmpltd_cmd.data[3] << 24 | 
                              cmpltd_cmd.data[2] << 16 | 
                              cmpltd_cmd.data[1] << 8  | 
                              cmpltd_cmd.data[0];
                        PRINT_DEBUG("\n%s: Cfg_Rd Cmplt_D %d bytes. Addr=0x%08x%08x val=0x%08x%08x bar=%d",__FUNCTION__, len, SPLIT64(addr), SPLIT64(val), bar_region);
                    }
                }
            }
        }
    }

    return val;
}


static void 
pcie_msix_notify(PCIDevice *d, uint16_t vector, bool trigger)
{
    PRINT_DEBUG("\n%s: notify vector %d trigger:%d \n", __FUNCTION__, vector, trigger);
    if (msix_enabled(d)) {
        if (trigger) {
            msix_notify(d, vector);
        }
    } else if (msi_enabled(d)) {
        if (trigger){
            msi_notify(d, vector);
        }
    }
}

static void 
pcie_msix_mem_write(void *opaque, uint32_t addr, uint32_t val, int len)
{ 
    uint32_t reg_base, reg_size;
    uint32_t base_addr;
    int bar_region = 0;  // Aries only has BAR 0/1 since its 64bit
    PCIE_MSIX_DEVState *d = opaque;
    bool write = false;

    reg_base  = d->dev.io_regions[bar_region].addr;
    reg_size  = d->dev.io_regions[bar_region].size;
    base_addr = d->dev.io_regions[bar_region].addr;

    PRINT_DEBUG("\n%s: mem_write BAR Base address=0x%x Addr=0x%x val=0x%x bar=%d\n", __FUNCTION__, base_addr, addr, val, bar_region);
    PRINT_DEBUG("\n%s: mem_write REAL address: 0x%x\n", __FUNCTION__, base_addr + addr);
    PRINT_DEBUG("\n%s: mem_write Addr=0x%x val=0x%x bar=%d\n", __FUNCTION__, addr-reg_base, val, bar_region);

    do_socket_transaction(opaque, write, addr, val, len);
}

static uint64_t
pcie_msix_mem_read(void *opaque, uint32_t addr, int len)
{  
    uint64_t val = -1;
    uint32_t reg_base, reg_size;
    uint32_t base_addr;
    int bar_region = 0;  // Aries only has BAR 0/1 since its 64bit
    PCIE_MSIX_DEVState *d = opaque;
    bool read = true;

    reg_base  = d->dev.io_regions[bar_region].addr;
    reg_size  = d->dev.io_regions[bar_region].size;
    base_addr = d->dev.io_regions[bar_region].addr;

    PRINT_DEBUG("\n%s: mem_read BAR Base address=0x%x Addr=0x%x val=0x%x bar=%d\n", __FUNCTION__, base_addr, addr, val, bar_region);
    PRINT_DEBUG("\n%s: mem_read REAL address: 0x%x\n", __FUNCTION__, base_addr + addr);    

    val = do_socket_transaction(opaque, read, addr, val, len);
    
    PRINT_DEBUG("\n%s: mem_read Addr=0x%x val=0x%x bar=%d\n", __FUNCTION__, addr-reg_base, val, bar_region);
    
    return val;
}

static void
pcie_msix_mem_writeb(void *opaque, target_phys_addr_t addr, uint32_t val)
{
    pcie_msix_mem_write(opaque, addr, val, 1);
}

static void
pcie_msix_mem_writew(void *opaque, target_phys_addr_t addr, uint32_t val)
{
    pcie_msix_mem_write(opaque, addr, val, 2);
}

static void
pcie_msix_mem_writel(void *opaque, target_phys_addr_t addr, uint32_t val)
{
    pcie_msix_mem_write(opaque, addr, val, 4);
}

static bool trans64wr_msl; // Most Significant LWord
static uint64_t trans64wr_val;
static bool trans64rd_msl; // Most Significant LWord
static uint64_t trans64rd_val;

static void
pcie_msix_mem_writed(void *opaque, target_phys_addr_t addr, uint32_t val)
{
    if (!trans64wr_msl) {
        trans64wr_val = val;
        trans64wr_msl = true;
    }
    else {
        uint64_t temp = val;
        trans64wr_val |= (temp << 32);
        pcie_msix_mem_write(opaque, addr, trans64wr_val, 8);
        trans64wr_msl = false;
    }
}

static uint32_t 
pcie_msix_mem_readb(void *opaque, target_phys_addr_t addr)
{
    return pcie_msix_mem_read(opaque, addr, 1);
}

static uint32_t 
pcie_msix_mem_readw(void *opaque, target_phys_addr_t addr)
{
    return pcie_msix_mem_read(opaque, addr, 2);
}

static uint32_t 
pcie_msix_mem_readl(void *opaque, target_phys_addr_t addr)
{
    return pcie_msix_mem_read(opaque, addr, 4);
}

static uint32_t 
pcie_msix_mem_readd(void *opaque, target_phys_addr_t addr)
{
    if (!trans64rd_msl) {
        trans64rd_val = pcie_msix_mem_read(opaque, addr, 8);
        trans64rd_msl = true;
        return (trans64rd_val & 0xffffffff);
    }
    else {
        trans64rd_msl = false;
        return (trans64rd_val >> 32);
    }
}

static CPUWriteMemoryFunc * const pcie_msix_mem_write_fn[] = {
    pcie_msix_mem_writeb, pcie_msix_mem_writew, pcie_msix_mem_writel, pcie_msix_mem_writed
};

static CPUReadMemoryFunc * const pcie_msix_mem_read_fn[] = {
    pcie_msix_mem_readb, pcie_msix_mem_readw, pcie_msix_mem_readl, pcie_msix_mem_readd
};

static void 
pcie_msix_mem_map(PCIDevice *dev, int region_num, 
                  pcibus_t addr, pcibus_t size, int type)
{
    uint32_t msix_mem_bar = 0;
    PCIE_MSIX_DEVState *d = DO_UPCAST(PCIE_MSIX_DEVState, dev, dev);
    cpu_register_physical_memory(addr, BAR_Regions[msix_mem_bar][0], d->mmio_index);
}

static uint32_t pcie_msix_read_config(PCIDevice *d,
                                     uint32_t address, int len)
{
    return pci_default_read_config(d, address, len);
}

static void pcie_msix_write_config(PCIDevice *d,
                                   uint32_t address, uint32_t val, int len)
{    
    pci_default_write_config(d, address, val, len);
    msix_write_config(d, address, val, len);
    /*pcie_cap_deverr_write_config(d, address, val, len);
    pcie_cap_flr_write_config(d, address, val, len);    
    pcie_aer_write_config_vbridge(d, address, val, len);
    pcie_aer_write_config(d, address, val, len);*/
}

static void pcie_msix_reset(DeviceState *qdev)
{
    PCIDevice *d = DO_UPCAST(PCIDevice, qdev, qdev);
    msix_reset(d);
    pcie_cap_deverr_reset(d);
}

static void pcie_msix_flr(PCIDevice *d)
{
    pci_device_reset(d);
}

static int pcie_msix_initfn(PCIDevice *pci_dev)
{
    PCIE_MSIX_DEVState *d = DO_UPCAST(PCIE_MSIX_DEVState, dev, pci_dev);
    PCIBridge *br = DO_UPCAST(PCIBridge, dev, pci_dev);
    PCIEPort *p = DO_UPCAST(PCIEPort, br, br);
    int rc;

    PRINT_DEBUG("%s: PCIE MSIX Device init...\n", __FUNCTION__);
    int msix_mem_bar = 0; // Since its a 64bit BAR, we take up BAR0 & BAR1

    d->vectors = 64;
    d->mmio_index = cpu_register_io_memory(pcie_msix_mem_read_fn, pcie_msix_mem_write_fn, d);

    rc = msix_init(&d->dev, d->vectors, msix_mem_bar, 0);  
    /*if (!rc) {
        PRINT_DEBUG("%s: Registering Bar %i as I/O BAR\n", __FUNCTION__, msix_mem_bar);
        pci_register_bar(&d->dev, msix_mem_bar, msix_bar_size(&d->dev), PCI_BASE_ADDRESS_SPACE_MEMORY, msix_mmio_map);
        PRINT_DEBUG("%s: MSI-X initialized (%d vectors)\n", __FUNCTION__, d->vectors);
    }
    else {
        PRINT_DEBUG("%s: MSI-X initialization failed!\n", __FUNCTION__);
        return rc;
    }*/
    
    // Activate the vectors
    /*for (i = 0; i < d->vectors; i++) {
        msix_vector_use(&d->dev, i);
    }*/
    
    pci_register_bar(&d->dev, msix_mem_bar, BAR_Regions[msix_mem_bar][0], BAR_Regions[msix_mem_bar][1], pcie_msix_mem_map);
    
    rc = pci_pcie_cap_init(&d->dev, PCIE_MSIX_EXP_OFFSET, PCI_EXP_TYPE_ENDPOINT, p->port);
    if (rc < 0) {
        return rc;
    }

    pci_set_word(&d->dev.config[PCI_VENDOR_ID],          g_cfg_init[0] & 0xffff);
    pci_set_word(&d->dev.config[PCI_DEVICE_ID],         (g_cfg_init[0] >> 16) & 0xffff);
    pci_set_word(&d->dev.config[PCI_COMMAND],            g_cfg_init[1] & 0xffff);
    pci_set_word(&d->dev.config[PCI_STATUS],            (g_cfg_init[1] >> 16) & 0xffff);
    pci_set_byte(&d->dev.config[PCI_REVISION_ID],        g_cfg_init[2] & 0xff);
    pci_set_byte(&d->dev.config[PCI_REVISION_ID+1],     (g_cfg_init[2] >> 8) & 0xff);
    pci_set_word(&d->dev.config[PCI_REVISION_ID+2],     (g_cfg_init[2] >> 16) & 0xffff);
    pci_set_byte(&d->dev.config[PCI_CACHE_LINE_SIZE],    g_cfg_init[3] & 0xff);
    pci_set_byte(&d->dev.config[PCI_LATENCY_TIMER],     (g_cfg_init[3] >> 8) & 0xff);
    pci_set_byte(&d->dev.config[PCI_HEADER_TYPE],       (g_cfg_init[3] >> 16) & 0xff);
    //pci_set_byte(&d->dev.config[PCI_BIST],              (g_cfg_init[3] >> 24) & 0xff);
    //pci_set_long(&d->dev.config[PCI_BASE_ADDRESS_0],     g_cfg_init[4]);
    //pci_set_long(&d->dev.config[PCI_BASE_ADDRESS_1],     g_cfg_init[5]);
    //pci_set_long(&d->dev.config[PCI_BASE_ADDRESS_2],     g_cfg_init[6]);
    //pci_set_long(&d->dev.config[PCI_BASE_ADDRESS_3],     g_cfg_init[7]);
    //pci_set_long(&d->dev.config[PCI_BASE_ADDRESS_4],     g_cfg_init[8]);
    //pci_set_long(&d->dev.config[PCI_BASE_ADDRESS_5],     g_cfg_init[9]);
    //pci_set_long(&d->dev.config[PCI_CARDBUS_CIS],        g_cfg_init[0xa]);
    pci_set_word(&d->dev.config[PCI_SUBSYSTEM_VENDOR_ID],g_cfg_init[0xb] & 0xffff);
    pci_set_word(&d->dev.config[PCI_SUBSYSTEM_ID],      (g_cfg_init[0xb] >> 16) & 0xffff);
    //pci_set_long(&d->dev.config[PCI_ROM_ADDRESS],        g_cfg_init[0xc]);
    //pci_set_byte(&d->dev.config[PCI_CAPABILITY_LIST],    g_cfg_init[0xd] & 0xff);
    pci_set_byte(&d->dev.config[PCI_INTERRUPT_LINE],     g_cfg_init[0xf] & 0xff);
    pci_set_byte(&d->dev.config[PCI_INTERRUPT_PIN],     (g_cfg_init[0xf] >> 8) & 0xff);
    pci_set_byte(&d->dev.config[PCI_MIN_GNT],           (g_cfg_init[0xf] >> 16) & 0xff);
    pci_set_byte(&d->dev.config[PCI_MAX_LAT],           (g_cfg_init[0xf] >> 24) & 0xff);

    // Power Capabilities
    pci_set_long(&d->dev.config[0x80],                   g_cfg_init[0x20]);
    pci_set_long(&d->dev.config[0x84],                   g_cfg_init[0x21]);
    pci_set_long(&d->dev.config[0x88],                   g_cfg_init[0x22]);
    pci_set_long(&d->dev.config[0x8c],                   g_cfg_init[0x23]);

    // MSI-X Capabilities
    pci_set_long(&d->dev.config[0xb0],                   g_cfg_init[0x2c]);
    pci_set_long(&d->dev.config[0xb4],                   g_cfg_init[0x2d]);
    pci_set_long(&d->dev.config[0xb8],                   g_cfg_init[0x2e]);
    pci_set_long(&d->dev.config[0xbc],                   g_cfg_init[0x2f]);

    pci_set_long(&d->dev.config[0xc0],                   g_cfg_init[0x30]);
    pci_set_long(&d->dev.config[0xc4],                   g_cfg_init[0x31]);
    pci_set_long(&d->dev.config[0xc8],                   g_cfg_init[0x32]);
    pci_set_long(&d->dev.config[0xcc],                   g_cfg_init[0x33]);
    pci_set_long(&d->dev.config[0xd0],                   g_cfg_init[0x34]);

    pci_set_long(&d->dev.config[0xd4],                   g_cfg_init[0x35]);
    pci_set_long(&d->dev.config[0xd8],                   g_cfg_init[0x36]);
    pci_set_long(&d->dev.config[0xdc],                   g_cfg_init[0x37]);
    pci_set_long(&d->dev.config[0xe0],                   g_cfg_init[0x38]);

    pci_set_long(&d->dev.config[0xe4],                   g_cfg_init[0x39]);
    pci_set_long(&d->dev.config[0xe8],                   g_cfg_init[0x3a]);
    pci_set_long(&d->dev.config[0xec],                   g_cfg_init[0x3b]);
    pci_set_long(&d->dev.config[0xf0],                   g_cfg_init[0x3c]);

    PRINT_DEBUG("%s: Init done\n", __FUNCTION__);
    return 0;
}

static int pcie_msix_exitfn(PCIDevice *pci_dev)
{
    pcie_aer_exit(pci_dev);
    msix_uninit(pci_dev);
    return pci_pcie_cap_exit(pci_dev);
}

PCIDevice* pcie_msix_init(PCIBus *bus)
{
    return pci_create_simple(bus, -1, "pcie_msix_device");
}

static const VMStateDescription vmstate_pcie_msix = {
    .name                   = "pcie-msix-device",
    .version_id             = 1,
    .minimum_version_id     = 1,
    .minimum_version_id_old = 1,
    .fields                 = (VMStateField[]) {
        VMSTATE_PCIE_DEVICE(dev, PCIE_MSIX_DEVState),
        VMSTATE_STRUCT(dev.aer_log, PCIE_MSIX_DEVState, 0, vmstate_pcie_aer_log, struct pcie_aer_log),
        VMSTATE_END_OF_LIST()
    }
};

static PCIDeviceInfo pcie_msix_info = {
    .qdev.name      = PCIE_MSIX_DEVICE,
    .qdev.desc      = "PCIE MSIX device template",
    .qdev.size      = sizeof(PCIE_MSIX_DEVState),
    .qdev.reset     = pcie_msix_reset,
    .qdev.vmsd      = &vmstate_pcie_msix,
    .is_express     = 1,
    .config_read    = pcie_msix_read_config,
    .config_write   = pcie_msix_write_config,
    .init           = pcie_msix_initfn,
    .exit           = pcie_msix_exitfn,
    .qdev.props     = (Property[]) {
        DEFINE_PROP_UINT32("vectors", PCIE_MSIX_DEVState, vectors, PCIE_MSIX_MSI_NR_VECTOR),        
        DEFINE_PROP_UINT16("aer_log_max", PCIESlot,
                            port.br.dev.aer_log.log_max,
                            PCIE_AER_LOG_MAX_DEFAULT),
        DEFINE_PROP_END_OF_LIST(),
    }
};

static void pcie_msix_register(void)
{
    pci_qdev_register(&pcie_msix_info);
}

device_init(pcie_msix_register);

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2010-12-27 18:13 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-12-21 20:24 [Qemu-devel] PCIe Transaction handling in Qemu Adnan Khaleel
2010-12-22 10:40 ` [Qemu-devel] " Isaku Yamahata
2010-12-22 11:24 ` [Qemu-devel] " Paul Brook
2010-12-27 18:12 [Qemu-devel] " Adnan Khaleel

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.