* [PATCH kvm-unit-tests v2 00/17] VT-d unit test
@ 2016-11-09 15:10 Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 01/17] x86/asm: add cpu_relax() Peter Xu
` (18 more replies)
0 siblings, 19 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
This is v2 of vt-d unit test series.
Patch "libcflat: add IS_ALIGNED() macro, and page sizes" is picked up
by Drew in the ARM GIC framework series, so please feel free to drop
it when needed.
v2:
- move cpu_relax patch to the beginning, and use them in all places
[Drew]
- replace all corresponding 256 into PCI_DEVFN_MAX, as well for
PCI_BAR_NUM [Drew]
- adding is_power_of_2() to replace ONE_BIT_ONLY() [Drew]
- add SZ_64K macro [Drew]
- declare pci_config_write[wb] in lib/asm-generic/pci-host-bridge.h [Alex]
- edu_reg_read/write() add "l" in func name [Drew]
- drop pci_set_master(), instead, provide pci_cmd_set_clr() [Drew]
- change return code into bool (always) for functions that apply
[Drew]
- keep old pci_find_dev() interface [Drew/Alex]
- use __raw_{read|write}*() for both vt-d and edu register read/writes
[Alex]
- remove pci_ prefix for all pci_dev fields [Drew]
- replace 0xff in cap_handlers[0xff] into (PCI_CAP_ID_MAX + 1) [Drew]
- make x86/unittest.cfg simpler by using q35 directly with eim=off
[Drew]
RFC -> v1:
- when init edu device fail, report_skip() rather than return error
[Radim]
- use asserts rather than "static bool inited" to avoid multiple init
of components (affects patch 1/2) [Drew]
- moving the first two patches out of the series [Drew]
- int vtd_init(), do not setup_idt() since smp_init() did it [Drew]
- when edu do not have MSI enabled, skip interrupt test [Radim]
- rename vtd_reg_*() into vtd_{read|write}[lq](), and move them to
header file [Drew]
- use PAGE_MASK when able [Drew]
- use "&" instead of "|" in intel-iommu init test (three places)
[Drew]
- use "vtd_init()" in unit test [Drew]
- mention that where intel-iommu.h comes from [Drew]
- re-written vtd_gcmd_or(), make it also work on even hardware [Drew]
- remove most of the oneline wrapper for VT-d registers, instead, use
vtd_{read|write}* with register names [Drew]
- remove useless BDF helpers [Drew]
- move edu device macros into header file [Drew]
- make edu_check_alive static inline [Drew]
- remove all useless wrappers in pci-edu.c [Drew]
- remove pci_dma_dir_t and all its users, instead, use "bool
from_device" [Drew]
- not use typedef for structs, to follow Linux/kvm-unit-tests coding
style [Drew]
- let pci_dev_init() clean and simple, then provide
pci_enable_defaults() for more complicated things [Drew]
- add one more patch to add intel-iommu test into x86/unittest [Radim]
- use 0x60 intr request instead of factorial to trigger edu device
interrupt [Drew]
- ...and some other changes I just forgot to note down...
Currently only a very small test scope is covered:
* VT-d init
* DMAR: 4 bytes copy
* IR: MSI
However this series could be a base point to add more test cases for
VT-d. The problem is, there are many IOMMU error conditions which are
very hard to be triggered in a real guest (IOMMU has merely no
interface for guest user, and it's totally running in the background).
This piece of work can be a start point if we want to do more
complicated things and play around with Intel IOMMU devices (also for
IOMMU regression tests).
Please review. Thanks,
=================
To run the test:
./x86/run ./x86/intel-iommu.flat \
-M q35,kernel-irqchip=split -global ioapic.version=0x20 \
-device intel-iommu,intremap=on -device edu
Sample output:
pxdev:kvm-unit-tests [new-iommu-ut]# ./iommu_run.sh
/root/git/qemu/bin/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
+-device pci-testdev -kernel ./x86/intel-iommu.flat -M q35,kernel-irqchip=split -global ioapic.version=0x20 -device intel-iommu,intremap=on -device edu
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
VT-d version: 0x10
cap: 0x0012008c22260206
ecap: 0x0000000000f00f1a
PASS: init status check
PASS: fault status check
PASS: QI enablement
DMAR table address: 0x0000000007ff9000
PASS: DMAR table setup
IR table address: 0x0000000007ff8000
PASS: IR table setup
PASS: DMAR enablement
PASS: IR enablement
PASS: DMAR support 39 bits address width
PASS: DMAR support huge pages
PCI: init dev 0x0020 BAR 0 [MEM] addr 0xfea00000
PCI detected cap 0x5
Detected MSI for device 0x20 offset 0x40
allocated vt-d root entry for PCI bus 0
allocated vt-d context entry for devfn 0x20
map 4K page IOVA 0x0 to 0x7ff7000 (sid=0x0020)
edu device DMA start TO addr 0x0 size 0x4 off 0x0
edu device DMA start FROM addr 0x4 size 0x4 off 0x0
PASS: DMAR 4B memcpy test
INTR: setup IRTE index 0
MSI: dev 0x20 init 64bit address: addr=0xfee00010, data=0x0
PASS: EDU factorial INTR test
Peter Xu (17):
x86/asm: add cpu_relax()
libcflat: introduce is_power_of_2()
x86: intel-iommu: add vt-d init test
libcflat: add IS_ALIGNED() macro, and page sizes
libcflat: moving MIN/MAX here
vm/page: provide PGDIR_OFFSET() macro
pci: introduce struct pci_dev
pci: provide pci_scan_bars()
x86/vmexit: leverage pci_scan_bars()
pci: provide pci_cmd_set_clr()
pci: provide pci_enable_defaults()
pci: add bdf helpers
pci: edu: introduce pci-edu helpers
x86: intel-iommu: add dmar test
pci: add msi support for 32/64bit address
x86: intel-iommu: add IR MSI test
x86/unittests: add intel-iommu test
lib/alloc.c | 3 -
lib/libcflat.h | 14 +++
lib/pci-edu.c | 73 ++++++++++++
lib/pci-edu.h | 83 +++++++++++++
lib/pci-host-generic.c | 9 +-
lib/pci-testdev.c | 10 +-
lib/pci.c | 154 ++++++++++++++++++++----
lib/pci.h | 39 ++++--
lib/x86/asm/barrier.h | 11 ++
lib/x86/asm/page.h | 3 +
lib/x86/intel-iommu.c | 313 +++++++++++++++++++++++++++++++++++++++++++++++++
lib/x86/intel-iommu.h | 142 ++++++++++++++++++++++
lib/x86/vm.c | 4 +-
x86/Makefile.common | 1 +
x86/Makefile.x86_64 | 2 +
x86/intel-iommu.c | 119 +++++++++++++++++++
x86/unittests.cfg | 7 ++
x86/vmexit.c | 27 ++---
18 files changed, 955 insertions(+), 59 deletions(-)
create mode 100644 lib/pci-edu.c
create mode 100644 lib/pci-edu.h
create mode 100644 lib/x86/intel-iommu.c
create mode 100644 lib/x86/intel-iommu.h
create mode 100644 x86/intel-iommu.c
--
2.7.4
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 01/17] x86/asm: add cpu_relax()
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 02/17] libcflat: introduce is_power_of_2() Peter Xu
` (17 subsequent siblings)
18 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
This will be useful to be put inside loops.
Suggested-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/x86/asm/barrier.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/lib/x86/asm/barrier.h b/lib/x86/asm/barrier.h
index 7c108bd..193fb4c 100644
--- a/lib/x86/asm/barrier.h
+++ b/lib/x86/asm/barrier.h
@@ -13,4 +13,15 @@
#define smp_rmb() barrier()
#define smp_wmb() barrier()
+/* REP NOP (PAUSE) is a good thing to insert into busy-wait loops. */
+static inline void rep_nop(void)
+{
+ asm volatile("rep; nop" ::: "memory");
+}
+
+static inline void cpu_relax(void)
+{
+ rep_nop();
+}
+
#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 02/17] libcflat: introduce is_power_of_2()
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 01/17] x86/asm: add cpu_relax() Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 18:20 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 03/17] x86: intel-iommu: add vt-d init test Peter Xu
` (16 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/libcflat.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/lib/libcflat.h b/lib/libcflat.h
index 72b1bf9..19bd0c6 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -103,4 +103,9 @@ do { \
} \
} while (0)
+static inline bool is_power_of_2(unsigned long n)
+{
+ return (n && !(n & (n - 1)));
+}
+
#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 03/17] x86: intel-iommu: add vt-d init test
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 01/17] x86/asm: add cpu_relax() Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 02/17] libcflat: introduce is_power_of_2() Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:09 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 04/17] libcflat: add IS_ALIGNED() macro, and page sizes Peter Xu
` (15 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Adding fundamental init test for Intel IOMMU. This includes basic
initialization of Intel IOMMU device, like DMAR (DMA Remapping),
IR (Interrupt Remapping), QI (Queue Invalidation), etc.
Further tests can use vtd_init() to initialize Intel IOMMU environment.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/x86/intel-iommu.c | 88 +++++++++++++++++++++++++++++++++++++
lib/x86/intel-iommu.h | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++
x86/Makefile.x86_64 | 2 +
x86/intel-iommu.c | 27 ++++++++++++
4 files changed, 235 insertions(+)
create mode 100644 lib/x86/intel-iommu.c
create mode 100644 lib/x86/intel-iommu.h
create mode 100644 x86/intel-iommu.c
diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
new file mode 100644
index 0000000..6f52697
--- /dev/null
+++ b/lib/x86/intel-iommu.c
@@ -0,0 +1,88 @@
+/*
+ * Intel IOMMU APIs
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ */
+
+#include "intel-iommu.h"
+
+#define VTD_RTA_MASK (PAGE_MASK)
+#define VTD_IRTA_MASK (PAGE_MASK)
+
+static uint64_t vtd_root_table(void)
+{
+ /* No extend root table support yet */
+ return vtd_readq(DMAR_RTADDR_REG) & VTD_RTA_MASK;
+}
+
+static uint64_t vtd_ir_table(void)
+{
+ return vtd_readq(DMAR_IRTA_REG) & VTD_IRTA_MASK;
+}
+
+static void vtd_gcmd_or(uint32_t cmd)
+{
+ uint32_t status;
+
+ /* We only allow set one bit for each time */
+ assert(is_power_of_2(cmd));
+
+ status = vtd_readl(DMAR_GSTS_REG);
+ vtd_writel(DMAR_GCMD_REG, status | cmd);
+
+ if (cmd & VTD_GCMD_ONE_SHOT_BITS) {
+ /* One-shot bits are taking effect immediately */
+ return;
+ }
+
+ /* Make sure IOMMU handled our command request */
+ while (!(vtd_readl(DMAR_GSTS_REG) & cmd))
+ cpu_relax();
+}
+
+static void vtd_dump_init_info(void)
+{
+ printf("VT-d version: 0x%x\n", vtd_readl(DMAR_VER_REG));
+ printf(" cap: 0x%016lx\n", vtd_readq(DMAR_CAP_REG));
+ printf(" ecap: 0x%016lx\n", vtd_readq(DMAR_ECAP_REG));
+}
+
+static void vtd_setup_root_table(void)
+{
+ void *root = alloc_page();
+
+ memset(root, 0, PAGE_SIZE);
+ vtd_writeq(DMAR_RTADDR_REG, virt_to_phys(root));
+ vtd_gcmd_or(VTD_GCMD_ROOT);
+ printf("DMAR table address: 0x%016lx\n", vtd_root_table());
+}
+
+static void vtd_setup_ir_table(void)
+{
+ void *root = alloc_page();
+
+ memset(root, 0, PAGE_SIZE);
+ /* 0xf stands for table size (2^(0xf+1) == 65536) */
+ vtd_writeq(DMAR_IRTA_REG, virt_to_phys(root) | 0xf);
+ vtd_gcmd_or(VTD_GCMD_IR_TABLE);
+ printf("IR table address: 0x%016lx\n", vtd_ir_table());
+}
+
+void vtd_init(void)
+{
+ setup_vm();
+ smp_init();
+
+ vtd_dump_init_info();
+ vtd_gcmd_or(VTD_GCMD_QI); /* Enable QI */
+ vtd_setup_root_table();
+ vtd_setup_ir_table();
+ vtd_gcmd_or(VTD_GCMD_DMAR); /* Enable DMAR */
+ vtd_gcmd_or(VTD_GCMD_IR); /* Enable IR */
+}
diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
new file mode 100644
index 0000000..d95d76c
--- /dev/null
+++ b/lib/x86/intel-iommu.h
@@ -0,0 +1,118 @@
+/*
+ * Intel IOMMU header
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ *
+ * (From include/linux/intel-iommu.h)
+ */
+
+#ifndef __INTEL_IOMMU_H__
+#define __INTEL_IOMMU_H__
+
+#include "libcflat.h"
+#include "vm.h"
+#include "isr.h"
+#include "smp.h"
+#include "desc.h"
+#include "asm/io.h"
+
+#define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
+
+/*
+ * Intel IOMMU register specification
+ */
+#define DMAR_VER_REG 0x0 /* Arch version supported by this IOMMU */
+#define DMAR_CAP_REG 0x8 /* Hardware supported capabilities */
+#define DMAR_CAP_REG_HI 0xc /* High 32-bit of DMAR_CAP_REG */
+#define DMAR_ECAP_REG 0x10 /* Extended capabilities supported */
+#define DMAR_ECAP_REG_HI 0X14
+#define DMAR_GCMD_REG 0x18 /* Global command */
+#define DMAR_GSTS_REG 0x1c /* Global status */
+#define DMAR_RTADDR_REG 0x20 /* Root entry table */
+#define DMAR_RTADDR_REG_HI 0X24
+#define DMAR_CCMD_REG 0x28 /* Context command */
+#define DMAR_CCMD_REG_HI 0x2c
+#define DMAR_FSTS_REG 0x34 /* Fault status */
+#define DMAR_FECTL_REG 0x38 /* Fault control */
+#define DMAR_FEDATA_REG 0x3c /* Fault event interrupt data */
+#define DMAR_FEADDR_REG 0x40 /* Fault event interrupt addr */
+#define DMAR_FEUADDR_REG 0x44 /* Upper address */
+#define DMAR_AFLOG_REG 0x58 /* Advanced fault control */
+#define DMAR_AFLOG_REG_HI 0X5c
+#define DMAR_PMEN_REG 0x64 /* Enable protected memory region */
+#define DMAR_PLMBASE_REG 0x68 /* PMRR low addr */
+#define DMAR_PLMLIMIT_REG 0x6c /* PMRR low limit */
+#define DMAR_PHMBASE_REG 0x70 /* PMRR high base addr */
+#define DMAR_PHMBASE_REG_HI 0X74
+#define DMAR_PHMLIMIT_REG 0x78 /* PMRR high limit */
+#define DMAR_PHMLIMIT_REG_HI 0x7c
+#define DMAR_IQH_REG 0x80 /* Invalidation queue head */
+#define DMAR_IQH_REG_HI 0X84
+#define DMAR_IQT_REG 0x88 /* Invalidation queue tail */
+#define DMAR_IQT_REG_HI 0X8c
+#define DMAR_IQA_REG 0x90 /* Invalidation queue addr */
+#define DMAR_IQA_REG_HI 0x94
+#define DMAR_ICS_REG 0x9c /* Invalidation complete status */
+#define DMAR_IRTA_REG 0xb8 /* Interrupt remapping table addr */
+#define DMAR_IRTA_REG_HI 0xbc
+#define DMAR_IECTL_REG 0xa0 /* Invalidation event control */
+#define DMAR_IEDATA_REG 0xa4 /* Invalidation event data */
+#define DMAR_IEADDR_REG 0xa8 /* Invalidation event address */
+#define DMAR_IEUADDR_REG 0xac /* Invalidation event address */
+#define DMAR_PQH_REG 0xc0 /* Page request queue head */
+#define DMAR_PQH_REG_HI 0xc4
+#define DMAR_PQT_REG 0xc8 /* Page request queue tail*/
+#define DMAR_PQT_REG_HI 0xcc
+#define DMAR_PQA_REG 0xd0 /* Page request queue address */
+#define DMAR_PQA_REG_HI 0xd4
+#define DMAR_PRS_REG 0xdc /* Page request status */
+#define DMAR_PECTL_REG 0xe0 /* Page request event control */
+#define DMAR_PEDATA_REG 0xe4 /* Page request event data */
+#define DMAR_PEADDR_REG 0xe8 /* Page request event address */
+#define DMAR_PEUADDR_REG 0xec /* Page event upper address */
+#define DMAR_MTRRCAP_REG 0x100 /* MTRR capability */
+#define DMAR_MTRRCAP_REG_HI 0x104
+#define DMAR_MTRRDEF_REG 0x108 /* MTRR default type */
+#define DMAR_MTRRDEF_REG_HI 0x10c
+
+#define VTD_GCMD_IR_TABLE (0x1000000)
+#define VTD_GCMD_IR (0x2000000)
+#define VTD_GCMD_QI (0x4000000)
+#define VTD_GCMD_WBF (0x8000000) /* Write Buffer Flush */
+#define VTD_GCMD_SFL (0x20000000) /* Set Fault Log */
+#define VTD_GCMD_ROOT (0x40000000)
+#define VTD_GCMD_DMAR (0x80000000)
+#define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
+ VTD_GCMD_SFL | VTD_GCMD_ROOT)
+
+#define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
+
+static inline void vtd_writel(unsigned int reg, uint32_t value)
+{
+ __raw_writel(value, vtd_reg(reg));
+}
+
+static inline void vtd_writeq(unsigned int reg, uint64_t value)
+{
+ __raw_writeq(value, vtd_reg(reg));
+}
+
+static inline uint32_t vtd_readl(unsigned int reg)
+{
+ return __raw_readl(vtd_reg(reg));
+}
+
+static inline uint64_t vtd_readq(unsigned int reg)
+{
+ return __raw_readq(vtd_reg(reg));
+}
+
+void vtd_init(void);
+
+#endif
diff --git a/x86/Makefile.x86_64 b/x86/Makefile.x86_64
index f82492b..3e2821e 100644
--- a/x86/Makefile.x86_64
+++ b/x86/Makefile.x86_64
@@ -4,6 +4,7 @@ ldarch = elf64-x86-64
CFLAGS += -mno-red-zone
cflatobjs += lib/x86/setjmp64.o
+cflatobjs += lib/x86/intel-iommu.o
tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
$(TEST_DIR)/emulator.flat $(TEST_DIR)/idt_test.flat \
@@ -14,6 +15,7 @@ tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
tests += $(TEST_DIR)/svm.flat
tests += $(TEST_DIR)/vmx.flat
tests += $(TEST_DIR)/tscdeadline_latency.flat
+tests += $(TEST_DIR)/intel-iommu.flat
include $(TEST_DIR)/Makefile.common
diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
new file mode 100644
index 0000000..f247913
--- /dev/null
+++ b/x86/intel-iommu.c
@@ -0,0 +1,27 @@
+/*
+ * Intel IOMMU unit test.
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ */
+
+#include "intel-iommu.h"
+
+int main(int argc, char *argv[])
+{
+ vtd_init();
+
+ report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
+ report("QI enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_QI);
+ report("DMAR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_ROOT);
+ report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
+ report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
+ report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
+
+ return report_summary();
+}
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 04/17] libcflat: add IS_ALIGNED() macro, and page sizes
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (2 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 03/17] x86: intel-iommu: add vt-d init test Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 05/17] libcflat: moving MIN/MAX here Peter Xu
` (14 subsequent siblings)
18 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
These macros will be useful to do page alignment checks.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/libcflat.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/libcflat.h b/lib/libcflat.h
index 19bd0c6..dd600b7 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -33,6 +33,12 @@
#define __ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask))
#define __ALIGN(x, a) __ALIGN_MASK(x, (typeof(x))(a) - 1)
#define ALIGN(x, a) __ALIGN((x), (a))
+#define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0)
+
+#define SZ_4K (0x1000)
+#define SZ_64K (0x10000)
+#define SZ_2M (0x200000)
+#define SZ_1G (0x40000000)
typedef uint8_t u8;
typedef int8_t s8;
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 05/17] libcflat: moving MIN/MAX here
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (3 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 04/17] libcflat: add IS_ALIGNED() macro, and page sizes Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 06/17] vm/page: provide PGDIR_OFFSET() macro Peter Xu
` (13 subsequent siblings)
18 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
That's something can be used outside alloc.c.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/alloc.c | 3 ---
lib/libcflat.h | 3 +++
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/alloc.c b/lib/alloc.c
index e1d7b8a..58af52b 100644
--- a/lib/alloc.c
+++ b/lib/alloc.c
@@ -7,9 +7,6 @@
#include "asm/spinlock.h"
#include "asm/io.h"
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-#define MAX(a, b) ((a) > (b) ? (a) : (b))
-
#define PHYS_ALLOC_NR_REGIONS 256
struct phys_alloc_region {
diff --git a/lib/libcflat.h b/lib/libcflat.h
index dd600b7..9b92e66 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -40,6 +40,9 @@
#define SZ_2M (0x200000)
#define SZ_1G (0x40000000)
+#define MIN(a, b) ((a) < (b) ? (a) : (b))
+#define MAX(a, b) ((a) > (b) ? (a) : (b))
+
typedef uint8_t u8;
typedef int8_t s8;
typedef uint16_t u16;
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 06/17] vm/page: provide PGDIR_OFFSET() macro
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (4 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 05/17] libcflat: moving MIN/MAX here Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 07/17] pci: introduce struct pci_dev Peter Xu
` (12 subsequent siblings)
18 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
This can be used in further patches.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/x86/asm/page.h | 3 +++
lib/x86/vm.c | 4 ++--
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/x86/asm/page.h b/lib/x86/asm/page.h
index 5044a49..c43bab2 100644
--- a/lib/x86/asm/page.h
+++ b/lib/x86/asm/page.h
@@ -41,5 +41,8 @@
#define PGDIR_MASK 1023
#endif
+#define PGDIR_BITS(lvl) (((lvl) - 1) * PGDIR_WIDTH + PAGE_SHIFT)
+#define PGDIR_OFFSET(va, lvl) (((va) >> PGDIR_BITS(lvl)) & PGDIR_MASK)
+
#endif /* !__ASSEMBLY__ */
#endif
diff --git a/lib/x86/vm.c b/lib/x86/vm.c
index f7e778b..cda4c5f 100644
--- a/lib/x86/vm.c
+++ b/lib/x86/vm.c
@@ -48,7 +48,7 @@ unsigned long *install_pte(unsigned long *cr3,
unsigned offset;
for (level = PAGE_LEVEL; level > pte_level; --level) {
- offset = ((unsigned long)virt >> ((level-1) * PGDIR_WIDTH + 12)) & PGDIR_MASK;
+ offset = PGDIR_OFFSET((unsigned long)virt, level);
if (!(pt[offset] & PT_PRESENT_MASK)) {
unsigned long *new_pt = pt_page;
if (!new_pt)
@@ -60,7 +60,7 @@ unsigned long *install_pte(unsigned long *cr3,
}
pt = phys_to_virt(pt[offset] & PT_ADDR_MASK);
}
- offset = ((unsigned long)virt >> ((level-1) * PGDIR_WIDTH + 12)) & PGDIR_MASK;
+ offset = PGDIR_OFFSET((unsigned long)virt, level);
pt[offset] = pte;
return &pt[offset];
}
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 07/17] pci: introduce struct pci_dev
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (5 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 06/17] vm/page: provide PGDIR_OFFSET() macro Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:21 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars() Peter Xu
` (11 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
To extend current PCI framework, we need a per-device struct to store
device specific information. Time to have a pci_dev struct. Most of the
current PCI APIs are converted to use this pci_dev object as the first
argument. Currently it only contains one field "bdf", which is the bdf
of current device.
For a few APIs like pci_config_*() ops or pci_find_dev(), I kept the old
interface (use PCI BDF value rather than "struct pci_dev") since they
can be used in a open context that without any specific PCI device.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci-host-generic.c | 9 +++++---
lib/pci-testdev.c | 10 +++++----
lib/pci.c | 57 ++++++++++++++++++++++++++++++--------------------
lib/pci.h | 24 ++++++++++++++-------
x86/vmexit.c | 18 +++++++++-------
5 files changed, 72 insertions(+), 46 deletions(-)
diff --git a/lib/pci-host-generic.c b/lib/pci-host-generic.c
index 6ac0f15..ea83d72 100644
--- a/lib/pci-host-generic.c
+++ b/lib/pci-host-generic.c
@@ -211,6 +211,7 @@ static bool pci_alloc_resource(pcidevaddr_t dev, int bar_num, u64 *addr)
bool pci_probe(void)
{
+ struct pci_dev pci_dev;
pcidevaddr_t dev;
u8 header;
u32 cmd;
@@ -225,6 +226,8 @@ bool pci_probe(void)
if (!pci_dev_exists(dev))
continue;
+ pci_dev_init(&pci_dev, dev);
+
/* We are only interested in normal PCI devices */
header = pci_config_readb(dev, PCI_HEADER_TYPE);
if ((header & PCI_HEADER_TYPE_MASK) != PCI_HEADER_TYPE_NORMAL)
@@ -236,15 +239,15 @@ bool pci_probe(void)
u64 addr;
if (pci_alloc_resource(dev, i, &addr)) {
- pci_bar_set_addr(dev, i, addr);
+ pci_bar_set_addr(&pci_dev, i, addr);
- if (pci_bar_is_memory(dev, i))
+ if (pci_bar_is_memory(&pci_dev, i))
cmd |= PCI_COMMAND_MEMORY;
else
cmd |= PCI_COMMAND_IO;
}
- if (pci_bar_is64(dev, i))
+ if (pci_bar_is64(&pci_dev, i))
i++;
}
diff --git a/lib/pci-testdev.c b/lib/pci-testdev.c
index ad482d3..7d298e6 100644
--- a/lib/pci-testdev.c
+++ b/lib/pci-testdev.c
@@ -163,9 +163,10 @@ static int pci_testdev_all(struct pci_test_dev_hdr *test,
int pci_testdev(void)
{
+ struct pci_dev pci_dev;
+ pcidevaddr_t dev;
phys_addr_t addr;
void __iomem *mem, *io;
- pcidevaddr_t dev;
int nr_tests = 0;
bool ret;
@@ -175,14 +176,15 @@ int pci_testdev(void)
"check QEMU '-device pci-testdev' parameter\n");
return -1;
}
+ pci_dev_init(&pci_dev, dev);
- ret = pci_bar_is_valid(dev, 0) && pci_bar_is_valid(dev, 1);
+ ret = pci_bar_is_valid(&pci_dev, 0) && pci_bar_is_valid(&pci_dev, 1);
assert(ret);
- addr = pci_bar_get_addr(dev, 0);
+ addr = pci_bar_get_addr(&pci_dev, 0);
mem = ioremap(addr, PAGE_SIZE);
- addr = pci_bar_get_addr(dev, 1);
+ addr = pci_bar_get_addr(&pci_dev, 1);
io = (void *)(unsigned long)addr;
nr_tests += pci_testdev_all(mem, &pci_testdev_mem_ops);
diff --git a/lib/pci.c b/lib/pci.c
index 6bd54cb..c0bbcba 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -13,16 +13,21 @@ bool pci_dev_exists(pcidevaddr_t dev)
pci_config_readw(dev, PCI_DEVICE_ID) != 0xffff);
}
+void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf)
+{
+ memset(dev, 0, sizeof(*dev));
+ dev->bdf = bdf;
+}
+
/* Scan bus look for a specific device. Only bus 0 scanned for now. */
pcidevaddr_t pci_find_dev(uint16_t vendor_id, uint16_t device_id)
{
pcidevaddr_t dev;
- for (dev = 0; dev < 256; ++dev) {
+ for (dev = 0; dev < PCI_DEVFN_MAX; ++dev)
if (pci_config_readw(dev, PCI_VENDOR_ID) == vendor_id &&
pci_config_readw(dev, PCI_DEVICE_ID) == device_id)
return dev;
- }
return PCIDEVADDR_INVALID;
}
@@ -33,12 +38,13 @@ uint32_t pci_bar_mask(uint32_t bar)
PCI_BASE_ADDRESS_IO_MASK : PCI_BASE_ADDRESS_MEM_MASK;
}
-uint32_t pci_bar_get(pcidevaddr_t dev, int bar_num)
+uint32_t pci_bar_get(struct pci_dev *dev, int bar_num)
{
- return pci_config_readl(dev, PCI_BASE_ADDRESS_0 + bar_num * 4);
+ return pci_config_readl(dev->bdf, PCI_BASE_ADDRESS_0 +
+ bar_num * 4);
}
-phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num)
+phys_addr_t pci_bar_get_addr(struct pci_dev *dev, int bar_num)
{
uint32_t bar = pci_bar_get(dev, bar_num);
uint32_t mask = pci_bar_mask(bar);
@@ -47,17 +53,18 @@ phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num)
if (pci_bar_is64(dev, bar_num))
addr |= (uint64_t)pci_bar_get(dev, bar_num + 1) << 32;
- return pci_translate_addr(dev, addr);
+ return pci_translate_addr(dev->bdf, addr);
}
-void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr)
+void pci_bar_set_addr(struct pci_dev *dev, int bar_num, phys_addr_t addr)
{
int off = PCI_BASE_ADDRESS_0 + bar_num * 4;
- pci_config_writel(dev, off, (uint32_t)addr);
+ pci_config_writel(dev->bdf, off, (uint32_t)addr);
if (pci_bar_is64(dev, bar_num))
- pci_config_writel(dev, off + 4, (uint32_t)(addr >> 32));
+ pci_config_writel(dev->bdf, off + 4,
+ (uint32_t)(addr >> 32));
}
/*
@@ -70,20 +77,21 @@ void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr)
* The following pci_bar_size_helper() and pci_bar_size() functions
* implement the algorithm.
*/
-static uint32_t pci_bar_size_helper(pcidevaddr_t dev, int bar_num)
+static uint32_t pci_bar_size_helper(struct pci_dev *dev, int bar_num)
{
int off = PCI_BASE_ADDRESS_0 + bar_num * 4;
+ uint16_t bdf = dev->bdf;
uint32_t bar, val;
- bar = pci_config_readl(dev, off);
- pci_config_writel(dev, off, ~0u);
- val = pci_config_readl(dev, off);
- pci_config_writel(dev, off, bar);
+ bar = pci_config_readl(bdf, off);
+ pci_config_writel(bdf, off, ~0u);
+ val = pci_config_readl(bdf, off);
+ pci_config_writel(bdf, off, bar);
return val;
}
-phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num)
+phys_addr_t pci_bar_size(struct pci_dev *dev, int bar_num)
{
uint32_t bar, size;
@@ -104,19 +112,19 @@ phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num)
}
}
-bool pci_bar_is_memory(pcidevaddr_t dev, int bar_num)
+bool pci_bar_is_memory(struct pci_dev *dev, int bar_num)
{
uint32_t bar = pci_bar_get(dev, bar_num);
return !(bar & PCI_BASE_ADDRESS_SPACE_IO);
}
-bool pci_bar_is_valid(pcidevaddr_t dev, int bar_num)
+bool pci_bar_is_valid(struct pci_dev *dev, int bar_num)
{
return pci_bar_get(dev, bar_num);
}
-bool pci_bar_is64(pcidevaddr_t dev, int bar_num)
+bool pci_bar_is64(struct pci_dev *dev, int bar_num)
{
uint32_t bar = pci_bar_get(dev, bar_num);
@@ -127,7 +135,7 @@ bool pci_bar_is64(pcidevaddr_t dev, int bar_num)
PCI_BASE_ADDRESS_MEM_TYPE_64;
}
-void pci_bar_print(pcidevaddr_t dev, int bar_num)
+void pci_bar_print(struct pci_dev *dev, int bar_num)
{
phys_addr_t size, start, end;
uint32_t bar;
@@ -187,6 +195,9 @@ static void pci_dev_print(pcidevaddr_t dev)
uint8_t subclass = pci_config_readb(dev, PCI_CLASS_DEVICE);
uint8_t class = pci_config_readb(dev, PCI_CLASS_DEVICE + 1);
int i;
+ struct pci_dev pci_dev;
+
+ pci_dev_init(&pci_dev, dev);
pci_dev_print_id(dev);
printf(" type %02x progif %02x class %02x subclass %02x\n",
@@ -196,12 +207,12 @@ static void pci_dev_print(pcidevaddr_t dev)
return;
for (i = 0; i < 6; i++) {
- if (pci_bar_size(dev, i)) {
+ if (pci_bar_size(&pci_dev, i)) {
printf("\t");
- pci_bar_print(dev, i);
+ pci_bar_print(&pci_dev, i);
printf("\n");
}
- if (pci_bar_is64(dev, i))
+ if (pci_bar_is64(&pci_dev, i))
i++;
}
}
@@ -210,7 +221,7 @@ void pci_print(void)
{
pcidevaddr_t dev;
- for (dev = 0; dev < 256; ++dev) {
+ for (dev = 0; dev < PCI_DEVFN_MAX; ++dev) {
if (pci_dev_exists(dev))
pci_dev_print(dev);
}
diff --git a/lib/pci.h b/lib/pci.h
index 30f5381..21f5a7b 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -15,6 +15,14 @@ enum {
PCIDEVADDR_INVALID = 0xffff,
};
+#define PCI_DEVFN_MAX (256)
+
+struct pci_dev {
+ uint16_t bdf;
+};
+
+void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
+
extern bool pci_probe(void);
extern void pci_print(void);
extern bool pci_dev_exists(pcidevaddr_t dev);
@@ -32,15 +40,15 @@ extern pcidevaddr_t pci_find_dev(uint16_t vendor_id, uint16_t device_id);
* It is expected the caller is aware of the device BAR layout and never
* tries to address the middle of a 64-bit register.
*/
-extern phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num);
-extern void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr);
-extern phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num);
-extern uint32_t pci_bar_get(pcidevaddr_t dev, int bar_num);
+extern phys_addr_t pci_bar_get_addr(struct pci_dev *dev, int bar_num);
+extern void pci_bar_set_addr(struct pci_dev *dev, int bar_num, phys_addr_t addr);
+extern phys_addr_t pci_bar_size(struct pci_dev *dev, int bar_num);
+extern uint32_t pci_bar_get(struct pci_dev *dev, int bar_num);
extern uint32_t pci_bar_mask(uint32_t bar);
-extern bool pci_bar_is64(pcidevaddr_t dev, int bar_num);
-extern bool pci_bar_is_memory(pcidevaddr_t dev, int bar_num);
-extern bool pci_bar_is_valid(pcidevaddr_t dev, int bar_num);
-extern void pci_bar_print(pcidevaddr_t dev, int bar_num);
+extern bool pci_bar_is64(struct pci_dev *dev, int bar_num);
+extern bool pci_bar_is_memory(struct pci_dev *dev, int bar_num);
+extern bool pci_bar_is_valid(struct pci_dev *dev, int bar_num);
+extern void pci_bar_print(struct pci_dev *dev, int bar_num);
extern void pci_dev_print_id(pcidevaddr_t dev);
int pci_testdev(void);
diff --git a/x86/vmexit.c b/x86/vmexit.c
index 2d99d5f..63fa070 100644
--- a/x86/vmexit.c
+++ b/x86/vmexit.c
@@ -372,7 +372,8 @@ int main(int ac, char **av)
struct fadt_descriptor_rev1 *fadt;
int i;
unsigned long membar = 0;
- pcidevaddr_t pcidev;
+ struct pci_dev pcidev;
+ int ret;
smp_init();
setup_vm();
@@ -385,21 +386,22 @@ int main(int ac, char **av)
pm_tmr_blk = fadt->pm_tmr_blk;
printf("PM timer port is %x\n", pm_tmr_blk);
- pcidev = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
- if (pcidev != PCIDEVADDR_INVALID) {
+ ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
+ if (ret != PCIDEVADDR_INVALID) {
+ pci_dev_init(&pcidev, ret);
for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
- if (!pci_bar_is_valid(pcidev, i)) {
+ if (!pci_bar_is_valid(&pcidev, i)) {
continue;
}
- if (pci_bar_is_memory(pcidev, i)) {
- membar = pci_bar_get_addr(pcidev, i);
+ if (pci_bar_is_memory(&pcidev, i)) {
+ membar = pci_bar_get_addr(&pcidev, i);
pci_test.memaddr = ioremap(membar, PAGE_SIZE);
} else {
- pci_test.iobar = pci_bar_get_addr(pcidev, i);
+ pci_test.iobar = pci_bar_get_addr(&pcidev, i);
}
}
printf("pci-testdev at 0x%x membar %lx iobar %x\n",
- pcidev, membar, pci_test.iobar);
+ pcidev.bdf, membar, pci_test.iobar);
}
for (i = 0; i < ARRAY_SIZE(tests); ++i)
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars()
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (6 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 07/17] pci: introduce struct pci_dev Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:24 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 09/17] x86/vmexit: leverage pci_scan_bars() Peter Xu
` (10 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Let's provide a more general way to scan PCI bars, rather than read the
config registers every time.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.c | 13 ++++++++++++-
lib/pci.h | 3 +++
2 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/lib/pci.c b/lib/pci.c
index c0bbcba..c063d53 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -206,7 +206,7 @@ static void pci_dev_print(pcidevaddr_t dev)
if ((header & PCI_HEADER_TYPE_MASK) != PCI_HEADER_TYPE_NORMAL)
return;
- for (i = 0; i < 6; i++) {
+ for (i = 0; i < PCI_BAR_NUM; i++) {
if (pci_bar_size(&pci_dev, i)) {
printf("\t");
pci_bar_print(&pci_dev, i);
@@ -226,3 +226,14 @@ void pci_print(void)
pci_dev_print(dev);
}
}
+
+void pci_scan_bars(struct pci_dev *dev)
+{
+ int i = 0;
+
+ for (i = 0; i < PCI_BAR_NUM; i++) {
+ if (!pci_bar_is_valid(dev, i))
+ continue;
+ dev->bar[i] = pci_bar_get_addr(dev, i);
+ }
+}
diff --git a/lib/pci.h b/lib/pci.h
index 21f5a7b..5ed4e11 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -15,13 +15,16 @@ enum {
PCIDEVADDR_INVALID = 0xffff,
};
+#define PCI_BAR_NUM (6)
#define PCI_DEVFN_MAX (256)
struct pci_dev {
uint16_t bdf;
+ phys_addr_t bar[PCI_BAR_NUM];
};
void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
+void pci_scan_bars(struct pci_dev *dev);
extern bool pci_probe(void);
extern void pci_print(void);
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 09/17] x86/vmexit: leverage pci_scan_bars()
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (7 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars() Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:27 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 10/17] pci: provide pci_cmd_set_clr() Peter Xu
` (9 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Since pci-testdev is a very specific device for QEMU, let's use the new
pci_scan_bars() helper, and selectively choose the bars we want.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.h | 2 ++
x86/vmexit.c | 17 ++++++-----------
2 files changed, 8 insertions(+), 11 deletions(-)
diff --git a/lib/pci.h b/lib/pci.h
index 5ed4e11..e452819 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -68,6 +68,8 @@ int pci_testdev(void);
* pci-testdev supports at least three types of tests (via mmio and
* portio BARs): no-eventfd, wildcard-eventfd and datamatch-eventfd
*/
+#define PCI_TESTDEV_BAR_MEM 0
+#define PCI_TESTDEV_BAR_IO 1
#define PCI_TESTDEV_NUM_BARS 2
#define PCI_TESTDEV_NUM_TESTS 3
diff --git a/x86/vmexit.c b/x86/vmexit.c
index 63fa070..a22f43f 100644
--- a/x86/vmexit.c
+++ b/x86/vmexit.c
@@ -389,17 +389,12 @@ int main(int ac, char **av)
ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
if (ret != PCIDEVADDR_INVALID) {
pci_dev_init(&pcidev, ret);
- for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
- if (!pci_bar_is_valid(&pcidev, i)) {
- continue;
- }
- if (pci_bar_is_memory(&pcidev, i)) {
- membar = pci_bar_get_addr(&pcidev, i);
- pci_test.memaddr = ioremap(membar, PAGE_SIZE);
- } else {
- pci_test.iobar = pci_bar_get_addr(&pcidev, i);
- }
- }
+ pci_scan_bars(&pcidev);
+ assert(pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_MEM));
+ assert(!pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_IO));
+ membar = pcidev.bar[PCI_TESTDEV_BAR_MEM];
+ pci_test.memaddr = ioremap(membar, PAGE_SIZE);
+ pci_test.iobar = pcidev.bar[PCI_TESTDEV_BAR_IO];
printf("pci-testdev at 0x%x membar %lx iobar %x\n",
pcidev.bdf, membar, pci_test.iobar);
}
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 10/17] pci: provide pci_cmd_set_clr()
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (8 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 09/17] x86/vmexit: leverage pci_scan_bars() Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:31 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 11/17] pci: provide pci_enable_defaults() Peter Xu
` (8 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Helper function to set/clear specific bit in PCI_COMMAND register.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.c | 12 ++++++++++++
lib/pci.h | 1 +
2 files changed, 13 insertions(+)
diff --git a/lib/pci.c b/lib/pci.c
index c063d53..fd17ea5 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -7,6 +7,18 @@
#include "pci.h"
#include "asm/pci.h"
+void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
+{
+ uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
+
+ /* No overlap is allowed */
+ assert((set & clr) == 0);
+ val |= set;
+ val &= ~clr;
+
+ pci_config_writew(dev->bdf, PCI_COMMAND, val);
+}
+
bool pci_dev_exists(pcidevaddr_t dev)
{
return (pci_config_readw(dev, PCI_VENDOR_ID) != 0xffff &&
diff --git a/lib/pci.h b/lib/pci.h
index e452819..de15086 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -25,6 +25,7 @@ struct pci_dev {
void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
void pci_scan_bars(struct pci_dev *dev);
+void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
extern bool pci_probe(void);
extern void pci_print(void);
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 11/17] pci: provide pci_enable_defaults()
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (9 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 10/17] pci: provide pci_cmd_set_clr() Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:33 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 12/17] pci: add bdf helpers Peter Xu
` (7 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Provide a function to do most of the common PCI init work.
Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.c | 8 ++++++++
lib/pci.h | 1 +
2 files changed, 9 insertions(+)
diff --git a/lib/pci.c b/lib/pci.c
index fd17ea5..971f02e 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -249,3 +249,11 @@ void pci_scan_bars(struct pci_dev *dev)
dev->bar[i] = pci_bar_get_addr(dev, i);
}
}
+
+int pci_enable_defaults(struct pci_dev *dev)
+{
+ pci_scan_bars(dev);
+ /* Enable device DMA operations */
+ pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
+ return 0;
+}
diff --git a/lib/pci.h b/lib/pci.h
index de15086..63f8a0a 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -26,6 +26,7 @@ struct pci_dev {
void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
void pci_scan_bars(struct pci_dev *dev);
void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
+int pci_enable_defaults(struct pci_dev *dev);
extern bool pci_probe(void);
extern void pci_print(void);
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 12/17] pci: add bdf helpers
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (10 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 11/17] pci: provide pci_enable_defaults() Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 13/17] pci: edu: introduce pci-edu helpers Peter Xu
` (6 subsequent siblings)
18 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/pci.h b/lib/pci.h
index 63f8a0a..0bdfd8c 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -18,6 +18,9 @@ enum {
#define PCI_BAR_NUM (6)
#define PCI_DEVFN_MAX (256)
+#define PCI_BDF_GET_DEVFN(x) ((x) & 0xff)
+#define PCI_BDF_GET_BUS(x) (((x) >> 8) & 0xff)
+
struct pci_dev {
uint16_t bdf;
phys_addr_t bar[PCI_BAR_NUM];
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 13/17] pci: edu: introduce pci-edu helpers
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (11 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 12/17] pci: add bdf helpers Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:45 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 14/17] x86: intel-iommu: add dmar test Peter Xu
` (5 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
QEMU edu device is a pci device that is originally written for
educational purpose, however it also suits for IOMMU unit test. Adding
helpers for this specific device to implement the device logic.
The device supports lots of functions, here only DMA operation is
supported.
The spec of the device can be found at:
https://github.com/qemu/qemu/blob/master/docs/specs/edu.txt
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci-edu.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++
lib/pci-edu.h | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 155 insertions(+)
create mode 100644 lib/pci-edu.c
create mode 100644 lib/pci-edu.h
diff --git a/lib/pci-edu.c b/lib/pci-edu.c
new file mode 100644
index 0000000..c9a8b0c
--- /dev/null
+++ b/lib/pci-edu.c
@@ -0,0 +1,73 @@
+/*
+ * Edu PCI device.
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ */
+
+#include "pci-edu.h"
+#include "asm/barrier.h"
+
+/* Return true if alive */
+static inline bool edu_check_alive(struct pci_edu_dev *dev)
+{
+ static uint32_t live_count = 1;
+ uint32_t value;
+
+ edu_reg_writel(dev, EDU_REG_ALIVE, live_count++);
+ value = edu_reg_readl(dev, EDU_REG_ALIVE);
+ return (live_count - 1 == ~value);
+}
+
+bool edu_init(struct pci_edu_dev *dev)
+{
+ pcidevaddr_t dev_addr;
+
+ dev_addr = pci_find_dev(PCI_VENDOR_ID_QEMU, PCI_DEVICE_ID_EDU);
+ if (dev_addr == PCIDEVADDR_INVALID)
+ return false;
+
+ pci_dev_init(&dev->pci_dev, dev_addr);
+ pci_enable_defaults(&dev->pci_dev);
+ assert(edu_check_alive(dev));
+ return true;
+}
+
+void edu_dma(struct pci_edu_dev *dev, iova_t iova,
+ size_t size, int dev_offset, bool from_device)
+{
+ uint64_t from, to;
+ uint32_t cmd = EDU_CMD_DMA_START;
+
+ assert(size <= EDU_DMA_SIZE_MAX);
+ assert(dev_offset < EDU_DMA_SIZE_MAX &&
+ dev_offset >= 0);
+
+ printf("edu device DMA start %s addr %p size 0x%lu off 0x%x\n",
+ from_device ? "FROM" : "TO",
+ (void *)iova, size, dev_offset);
+
+ if (from_device) {
+ from = dev_offset + EDU_DMA_START;
+ to = iova;
+ cmd |= EDU_CMD_DMA_FROM;
+ } else {
+ from = iova;
+ to = EDU_DMA_START + dev_offset;
+ cmd |= EDU_CMD_DMA_TO;
+ }
+
+ edu_reg_writeq(dev, EDU_REG_DMA_SRC, from);
+ edu_reg_writeq(dev, EDU_REG_DMA_DST, to);
+ edu_reg_writeq(dev, EDU_REG_DMA_COUNT, size);
+ edu_reg_writel(dev, EDU_REG_DMA_CMD, cmd);
+
+ /* Wait until DMA finished */
+ while (edu_reg_readl(dev, EDU_REG_DMA_CMD) & EDU_CMD_DMA_START)
+ cpu_relax();
+}
diff --git a/lib/pci-edu.h b/lib/pci-edu.h
new file mode 100644
index 0000000..af457df
--- /dev/null
+++ b/lib/pci-edu.h
@@ -0,0 +1,82 @@
+/*
+ * Edu PCI device header.
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ *
+ * Edu device is a virtualized device in QEMU. Please refer to
+ * docs/specs/edu.txt in QEMU repository for EDU device manual.
+ */
+#ifndef __PCI_EDU_H__
+#define __PCI_EDU_H__
+
+#include "pci.h"
+#include "asm/io.h"
+
+#define PCI_VENDOR_ID_QEMU (0x1234)
+#define PCI_DEVICE_ID_EDU (0x11e8)
+
+/* The only bar used by EDU device */
+#define EDU_BAR_MEM (0)
+#define EDU_MAGIC (0xed)
+#define EDU_VERSION (0x100)
+#define EDU_DMA_BUF_SIZE (1 << 20)
+#define EDU_INPUT_BUF_SIZE (256)
+
+#define EDU_REG_ID (0x0)
+#define EDU_REG_ALIVE (0x4)
+#define EDU_REG_FACTORIAL (0x8)
+#define EDU_REG_STATUS (0x20)
+#define EDU_REG_DMA_SRC (0x80)
+#define EDU_REG_DMA_DST (0x88)
+#define EDU_REG_DMA_COUNT (0x90)
+#define EDU_REG_DMA_CMD (0x98)
+
+#define EDU_CMD_DMA_START (0x01)
+#define EDU_CMD_DMA_FROM (0x02)
+#define EDU_CMD_DMA_TO (0x00)
+
+#define EDU_STATUS_FACTORIAL (0x1)
+#define EDU_STATUS_INT_ENABLE (0x80)
+
+#define EDU_DMA_START (0x40000)
+#define EDU_DMA_SIZE_MAX (4096)
+
+struct pci_edu_dev {
+ struct pci_dev pci_dev;
+};
+
+#define edu_reg(d, r) (volatile void *)((d)->pci_dev.bar[EDU_BAR_MEM] + (r))
+
+static inline uint64_t edu_reg_readq(struct pci_edu_dev *dev, int reg)
+{
+ return __raw_readq(edu_reg(dev, reg));
+}
+
+static inline uint32_t edu_reg_readl(struct pci_edu_dev *dev, int reg)
+{
+ return __raw_readl(edu_reg(dev, reg));
+}
+
+static inline void edu_reg_writeq(struct pci_edu_dev *dev, int reg,
+ uint64_t val)
+{
+ __raw_writeq(val, edu_reg(dev, reg));
+}
+
+static inline void edu_reg_writel(struct pci_edu_dev *dev, int reg,
+ uint32_t val)
+{
+ __raw_writel(val, edu_reg(dev, reg));
+}
+
+bool edu_init(struct pci_edu_dev *dev);
+void edu_dma(struct pci_edu_dev *dev, iova_t iova,
+ size_t size, int dev_offset, bool from_device);
+
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 14/17] x86: intel-iommu: add dmar test
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (12 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 13/17] pci: edu: introduce pci-edu helpers Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 19:53 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 15/17] pci: add msi support for 32/64bit address Peter Xu
` (4 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
DMAR test is based on QEMU edu device. A 4B DMA memory copy is carried
out as the simplest DMAR test.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.h | 2 +
lib/x86/intel-iommu.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++
lib/x86/intel-iommu.h | 23 +++++++++
x86/Makefile.common | 1 +
x86/intel-iommu.c | 50 +++++++++++++++++++
5 files changed, 207 insertions(+)
diff --git a/lib/pci.h b/lib/pci.h
index 0bdfd8c..c4fef98 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -31,6 +31,8 @@ void pci_scan_bars(struct pci_dev *dev);
void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
int pci_enable_defaults(struct pci_dev *dev);
+typedef phys_addr_t iova_t;
+
extern bool pci_probe(void);
extern void pci_print(void);
extern bool pci_dev_exists(pcidevaddr_t dev);
diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
index 6f52697..912a379 100644
--- a/lib/x86/intel-iommu.c
+++ b/lib/x86/intel-iommu.c
@@ -11,6 +11,42 @@
*/
#include "intel-iommu.h"
+#include "libcflat.h"
+
+/*
+ * VT-d in QEMU currently only support 39 bits address width, which is
+ * 3-level translation.
+ */
+#define VTD_PAGE_LEVEL (3)
+#define VTD_CE_AW_39BIT (0x1)
+
+typedef uint64_t vtd_pte_t;
+
+struct vtd_root_entry {
+ /* Quad 1 */
+ uint64_t present:1;
+ uint64_t __reserved:11;
+ uint64_t context_table_p:52;
+ /* Quad 2 */
+ uint64_t __reserved_2;
+} __attribute__ ((packed));
+typedef struct vtd_root_entry vtd_re_t;
+
+struct vtd_context_entry {
+ /* Quad 1 */
+ uint64_t present:1;
+ uint64_t disable_fault_report:1;
+ uint64_t trans_type:2;
+ uint64_t __reserved:8;
+ uint64_t slptptr:52;
+ /* Quad 2 */
+ uint64_t addr_width:3;
+ uint64_t __ignore:4;
+ uint64_t __reserved_2:1;
+ uint64_t domain_id:16;
+ uint64_t __reserved_3:40;
+} __attribute__ ((packed));
+typedef struct vtd_context_entry vtd_ce_t;
#define VTD_RTA_MASK (PAGE_MASK)
#define VTD_IRTA_MASK (PAGE_MASK)
@@ -74,6 +110,101 @@ static void vtd_setup_ir_table(void)
printf("IR table address: 0x%016lx\n", vtd_ir_table());
}
+static void vtd_install_pte(vtd_pte_t *root, iova_t iova,
+ phys_addr_t pa, int level_target)
+{
+ int level;
+ unsigned int offset;
+ void *page;
+
+ for (level = VTD_PAGE_LEVEL; level > level_target; level--) {
+ offset = PGDIR_OFFSET(iova, level);
+ if (!(root[offset] & VTD_PTE_RW)) {
+ page = alloc_page();
+ memset(page, 0, PAGE_SIZE);
+ root[offset] = virt_to_phys(page) | VTD_PTE_RW;
+ }
+ root = (uint64_t *)(root[offset] & VTD_PTE_ADDR);
+ }
+
+ offset = PGDIR_OFFSET(iova, level);
+ root[offset] = pa | VTD_PTE_RW;
+ if (level != 1)
+ /* This is huge page */
+ root[offset] |= VTD_PTE_HUGE;
+}
+
+#define VTD_FETCH_VIRT_ADDR(x) \
+ ((void *)(((uint64_t)phys_to_virt(x)) >> PAGE_SHIFT))
+
+/**
+ * vtd_map_range: setup IO address mapping for specific memory range
+ *
+ * @sid: source ID of the device to setup
+ * @iova: start IO virtual address
+ * @pa: start physical address
+ * @size: size of the mapping area
+ */
+void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
+{
+ uint8_t bus_n, devfn;
+ void *slptptr;
+ vtd_ce_t *ce;
+ vtd_re_t *re = phys_to_virt(vtd_root_table());
+
+ assert(IS_ALIGNED(iova, SZ_4K));
+ assert(IS_ALIGNED(pa, SZ_4K));
+ assert(IS_ALIGNED(size, SZ_4K));
+
+ bus_n = PCI_BDF_GET_BUS(sid);
+ devfn = PCI_BDF_GET_DEVFN(sid);
+
+ /* Point to the correct root entry */
+ re += bus_n;
+
+ if (!re->present) {
+ ce = alloc_page();
+ memset(ce, 0, PAGE_SIZE);
+ memset(re, 0, sizeof(*re));
+ re->context_table_p = virt_to_phys(ce) >> PAGE_SHIFT;
+ re->present = 1;
+ printf("allocated vt-d root entry for PCI bus %d\n",
+ bus_n);
+ } else
+ ce = VTD_FETCH_VIRT_ADDR(re->context_table_p);
+
+ /* Point to the correct context entry */
+ ce += devfn;
+
+ if (!ce->present) {
+ slptptr = alloc_page();
+ memset(slptptr, 0, PAGE_SIZE);
+ memset(ce, 0, sizeof(*ce));
+ /* To make it simple, domain ID is the same as SID */
+ ce->domain_id = sid;
+ /* We only test 39 bits width case (3-level paging) */
+ ce->addr_width = VTD_CE_AW_39BIT;
+ ce->slptptr = virt_to_phys(slptptr) >> PAGE_SHIFT;
+ ce->trans_type = VTD_CONTEXT_TT_MULTI_LEVEL;
+ ce->present = 1;
+ /* No error reporting yet */
+ ce->disable_fault_report = 1;
+ printf("allocated vt-d context entry for devfn 0x%x\n",
+ devfn);
+ } else
+ slptptr = VTD_FETCH_VIRT_ADDR(ce->slptptr);
+
+ while (size) {
+ /* TODO: currently we only map 4K pages (level = 1) */
+ printf("map 4K page IOVA 0x%lx to 0x%lx (sid=0x%04x)\n",
+ iova, pa, sid);
+ vtd_install_pte(slptptr, iova, pa, 1);
+ size -= PAGE_SIZE;
+ iova += PAGE_SIZE;
+ pa += PAGE_SIZE;
+ }
+}
+
void vtd_init(void)
{
setup_vm();
diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
index d95d76c..806d9fa 100644
--- a/lib/x86/intel-iommu.h
+++ b/lib/x86/intel-iommu.h
@@ -20,6 +20,7 @@
#include "isr.h"
#include "smp.h"
#include "desc.h"
+#include "pci.h"
#include "asm/io.h"
#define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
@@ -91,6 +92,27 @@
#define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
VTD_GCMD_SFL | VTD_GCMD_ROOT)
+/* Supported Adjusted Guest Address Widths */
+#define VTD_CAP_SAGAW_SHIFT 8
+ /* 39-bit AGAW, 3-level page-table */
+#define VTD_CAP_SAGAW_39bit (0x2ULL << VTD_CAP_SAGAW_SHIFT)
+ /* 48-bit AGAW, 4-level page-table */
+#define VTD_CAP_SAGAW_48bit (0x4ULL << VTD_CAP_SAGAW_SHIFT)
+#define VTD_CAP_SAGAW VTD_CAP_SAGAW_39bit
+
+/* Both 1G/2M huge pages */
+#define VTD_CAP_SLLPS ((1ULL << 34) | (1ULL << 35))
+
+#define VTD_CONTEXT_TT_MULTI_LEVEL 0
+#define VTD_CONTEXT_TT_DEV_IOTLB 1
+#define VTD_CONTEXT_TT_PASS_THROUGH 2
+
+#define VTD_PTE_R (1 << 0)
+#define VTD_PTE_W (1 << 1)
+#define VTD_PTE_RW (VTD_PTE_R | VTD_PTE_W)
+#define VTD_PTE_ADDR GENMASK_ULL(51, 12)
+#define VTD_PTE_HUGE (1 << 7)
+
#define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
static inline void vtd_writel(unsigned int reg, uint32_t value)
@@ -114,5 +136,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
}
void vtd_init(void);
+void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
#endif
diff --git a/x86/Makefile.common b/x86/Makefile.common
index 356d879..1dad18b 100644
--- a/x86/Makefile.common
+++ b/x86/Makefile.common
@@ -3,6 +3,7 @@
all: test_cases
cflatobjs += lib/pci.o
+cflatobjs += lib/pci-edu.o
cflatobjs += lib/x86/io.o
cflatobjs += lib/x86/smp.o
cflatobjs += lib/x86/vm.o
diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
index f247913..755ccb1 100644
--- a/x86/intel-iommu.c
+++ b/x86/intel-iommu.c
@@ -11,9 +11,49 @@
*/
#include "intel-iommu.h"
+#include "pci-edu.h"
+
+#define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
+
+void vtd_test_dmar(struct pci_edu_dev *dev)
+{
+ void *page = alloc_page();
+
+#define DMA_TEST_WORD (0x12345678)
+ /* Modify the first 4 bytes of the page */
+ *(uint32_t *)page = DMA_TEST_WORD;
+
+ /*
+ * Map the newly allocated page into IOVA address 0 (size 4K)
+ * of the device address space. Root entry and context entry
+ * will be automatically created when needed.
+ */
+ vtd_map_range(dev->pci_dev.bdf, 0, virt_to_phys(page), PAGE_SIZE);
+
+ /*
+ * DMA the first 4 bytes of the page to EDU device buffer
+ * offset 0.
+ */
+ edu_dma(dev, 0, 4, 0, false);
+ /*
+ * DMA the first 4 bytes of EDU device buffer into the page
+ * with offset 4 (so it'll be using 4-7 bytes).
+ */
+ edu_dma(dev, 4, 4, 0, true);
+
+ /*
+ * Check data match between 0-3 bytes and 4-7 bytes of the
+ * page.
+ */
+ report(VTD_TEST_DMAR_4B, *((uint32_t *)page + 1) == DMA_TEST_WORD);
+
+ free_page(page);
+}
int main(int argc, char *argv[])
{
+ struct pci_edu_dev dev;
+
vtd_init();
report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
@@ -22,6 +62,16 @@ int main(int argc, char *argv[])
report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
+ report("DMAR support 39 bits address width",
+ vtd_readq(DMAR_CAP_REG) & VTD_CAP_SAGAW);
+ report("DMAR support huge pages", vtd_readq(DMAR_CAP_REG) & VTD_CAP_SLLPS);
+
+ if (!edu_init(&dev)) {
+ printf("Please specify \"-device edu\" to do "
+ "further IOMMU tests.\n");
+ report_skip(VTD_TEST_DMAR_4B);
+ } else
+ vtd_test_dmar(&dev);
return report_summary();
}
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 15/17] pci: add msi support for 32/64bit address
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (13 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 14/17] x86: intel-iommu: add dmar test Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 20:10 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 16/17] x86: intel-iommu: add IR MSI test Peter Xu
` (3 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
pci_cap_walk() is provided to allow walk through all the capability bits
for a specific PCI device. If a cap handler is provided, it'll be
triggered if the cap is detected along the cap walk.
MSI cap handler is the first one supported. We can add more cap handler
in the future.
Meanwhile, pci_setup_msi() API is provided to support basic 32/64 bit
address MSI setup.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
lib/pci.h | 3 +++
2 files changed, 67 insertions(+)
diff --git a/lib/pci.c b/lib/pci.c
index 971f02e..5b474f2 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -7,6 +7,69 @@
#include "pci.h"
#include "asm/pci.h"
+typedef void (*pci_cap_handler)(struct pci_dev *dev, int cap_offset);
+
+static void pci_cap_msi_handler(struct pci_dev *dev, int cap_offset)
+{
+ printf("Detected MSI for device 0x%x offset 0x%x\n",
+ dev->bdf, cap_offset);
+ dev->msi_offset = cap_offset;
+}
+
+static pci_cap_handler cap_handlers[PCI_CAP_ID_MAX + 1] = {
+ [PCI_CAP_ID_MSI] = pci_cap_msi_handler,
+};
+
+void pci_cap_walk(struct pci_dev *dev)
+{
+ uint8_t cap_offset;
+ uint8_t cap_id;
+
+ cap_offset = pci_config_readb(dev->bdf, PCI_CAPABILITY_LIST);
+ while (cap_offset) {
+ cap_id = pci_config_readb(dev->bdf, cap_offset);
+ printf("PCI detected cap 0x%x\n", cap_id);
+ if (cap_handlers[cap_id])
+ cap_handlers[cap_id](dev, cap_offset);
+ cap_offset = pci_config_readb(dev->bdf, cap_offset + 1);
+ }
+}
+
+bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data)
+{
+ uint16_t msi_control;
+ uint16_t offset;
+ pcidevaddr_t addr = dev->bdf;
+
+ assert(dev);
+
+ if (!dev->msi_offset) {
+ printf("MSI: dev 0x%x does not support MSI.\n", addr);
+ return false;
+ }
+
+ offset = dev->msi_offset;
+ msi_control = pci_config_readw(addr, offset + PCI_MSI_FLAGS);
+ pci_config_writel(addr, offset + PCI_MSI_ADDRESS_LO,
+ msi_addr & 0xffffffff);
+
+ if (msi_control & PCI_MSI_FLAGS_64BIT) {
+ pci_config_writel(addr, offset + PCI_MSI_ADDRESS_HI,
+ (uint32_t)(msi_addr >> 32));
+ pci_config_writel(addr, offset + PCI_MSI_DATA_64, msi_data);
+ printf("MSI: dev 0x%x init 64bit address: ", addr);
+ } else {
+ pci_config_writel(addr, offset + PCI_MSI_DATA_32, msi_data);
+ printf("MSI: dev 0x%x init 32bit address: ", addr);
+ }
+ printf("addr=0x%lx, data=0x%x\n", msi_addr, msi_data);
+
+ msi_control |= PCI_MSI_FLAGS_ENABLE;
+ pci_config_writew(addr, offset + PCI_MSI_FLAGS, msi_control);
+
+ return true;
+}
+
void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
{
uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
@@ -255,5 +318,6 @@ int pci_enable_defaults(struct pci_dev *dev)
pci_scan_bars(dev);
/* Enable device DMA operations */
pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
+ pci_cap_walk(dev);
return 0;
}
diff --git a/lib/pci.h b/lib/pci.h
index c4fef98..a5a1454 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -23,13 +23,16 @@ enum {
struct pci_dev {
uint16_t bdf;
+ uint16_t msi_offset;
phys_addr_t bar[PCI_BAR_NUM];
};
void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
void pci_scan_bars(struct pci_dev *dev);
void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
+void pci_cap_walk(struct pci_dev *dev);
int pci_enable_defaults(struct pci_dev *dev);
+bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data);
typedef phys_addr_t iova_t;
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 16/17] x86: intel-iommu: add IR MSI test
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (14 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 15/17] pci: add msi support for 32/64bit address Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 20:18 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 17/17] x86/unittests: add intel-iommu test Peter Xu
` (2 subsequent siblings)
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
First of all, vtd_setup_msi() is provided. It setup IRTE entries,
meanwhile, setup PCI device MSI vectors corresponding to VT-d spec.
Meanwhile, IR MSI test is added to intel IOMMU unit test. The basic IR
test is carried out by a edu INTR raise request. When write to the intr
raise register, interrupt will be generated. Type of interrupt will
depend on the setup (either INTx or MSI).
Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci-edu.h | 1 +
lib/x86/intel-iommu.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++
lib/x86/intel-iommu.h | 1 +
x86/intel-iommu.c | 44 +++++++++++++++++++++++-
4 files changed, 139 insertions(+), 1 deletion(-)
diff --git a/lib/pci-edu.h b/lib/pci-edu.h
index af457df..5be278c 100644
--- a/lib/pci-edu.h
+++ b/lib/pci-edu.h
@@ -32,6 +32,7 @@
#define EDU_REG_ALIVE (0x4)
#define EDU_REG_FACTORIAL (0x8)
#define EDU_REG_STATUS (0x20)
+#define EDU_REG_INTR_RAISE (0x60)
#define EDU_REG_DMA_SRC (0x80)
#define EDU_REG_DMA_DST (0x88)
#define EDU_REG_DMA_COUNT (0x90)
diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
index 912a379..1d5f4b2 100644
--- a/lib/x86/intel-iommu.c
+++ b/lib/x86/intel-iommu.c
@@ -12,6 +12,7 @@
#include "intel-iommu.h"
#include "libcflat.h"
+#include "pci.h"
/*
* VT-d in QEMU currently only support 39 bits address width, which is
@@ -48,6 +49,26 @@ struct vtd_context_entry {
} __attribute__ ((packed));
typedef struct vtd_context_entry vtd_ce_t;
+struct vtd_irte {
+ uint32_t present:1;
+ uint32_t fault_disable:1; /* Fault Processing Disable */
+ uint32_t dest_mode:1; /* Destination Mode */
+ uint32_t redir_hint:1; /* Redirection Hint */
+ uint32_t trigger_mode:1; /* Trigger Mode */
+ uint32_t delivery_mode:3; /* Delivery Mode */
+ uint32_t __avail:4; /* Available spaces for software */
+ uint32_t __reserved_0:3; /* Reserved 0 */
+ uint32_t irte_mode:1; /* IRTE Mode */
+ uint32_t vector:8; /* Interrupt Vector */
+ uint32_t __reserved_1:8; /* Reserved 1 */
+ uint32_t dest_id; /* Destination ID */
+ uint16_t source_id:16; /* Source-ID */
+ uint64_t sid_q:2; /* Source-ID Qualifier */
+ uint64_t sid_vtype:2; /* Source-ID Validation Type */
+ uint64_t __reserved_2:44; /* Reserved 2 */
+} __attribute__ ((packed));
+typedef struct vtd_irte vtd_irte_t;
+
#define VTD_RTA_MASK (PAGE_MASK)
#define VTD_IRTA_MASK (PAGE_MASK)
@@ -205,6 +226,79 @@ void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
}
}
+static uint16_t vtd_intr_index_alloc(void)
+{
+ static int index_ctr = 0;
+ assert(index_ctr < 65535);
+ return index_ctr++;
+}
+
+static void vtd_setup_irte(struct pci_dev *dev, vtd_irte_t *irte,
+ int vector, int dest_id)
+{
+ assert(sizeof(vtd_irte_t) == 16);
+ memset(irte, 0, sizeof(*irte));
+ irte->fault_disable = 1;
+ irte->dest_mode = 0; /* physical */
+ irte->trigger_mode = 0; /* edge */
+ irte->delivery_mode = 0; /* fixed */
+ irte->irte_mode = 0; /* remapped */
+ irte->vector = vector;
+ irte->dest_id = dest_id;
+ irte->source_id = dev->bdf;
+ irte->sid_q = 0;
+ irte->sid_vtype = 1; /* full-sid verify */
+ irte->present = 1;
+}
+
+struct vtd_msi_addr {
+ uint32_t __dont_care:2;
+ uint32_t handle_15:1; /* handle[15] */
+ uint32_t shv:1;
+ uint32_t interrupt_format:1;
+ uint32_t handle_0_14:15; /* handle[0:14] */
+ uint32_t head:12; /* 0xfee */
+ uint32_t addr_hi; /* not used except with x2apic */
+} __attribute__ ((packed));
+typedef struct vtd_msi_addr vtd_msi_addr_t;
+
+struct vtd_msi_data {
+ uint16_t __reserved;
+ uint16_t subhandle;
+} __attribute__ ((packed));
+typedef struct vtd_msi_data vtd_msi_data_t;
+
+/**
+ * vtd_setup_msi - setup MSI message for a device
+ *
+ * @dev: PCI device to setup MSI
+ * @vector: interrupt vector
+ * @dest_id: destination processor
+ */
+bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id)
+{
+ vtd_msi_data_t msi_data = {};
+ vtd_msi_addr_t msi_addr = {};
+ vtd_irte_t *irte = phys_to_virt(vtd_ir_table());
+ uint16_t index = vtd_intr_index_alloc();
+
+ assert(sizeof(vtd_msi_addr_t) == 8);
+ assert(sizeof(vtd_msi_data_t) == 4);
+
+ printf("INTR: setup IRTE index %d\n", index);
+ vtd_setup_irte(dev, irte + index, vector, dest_id);
+
+ msi_addr.handle_15 = index >> 15 & 1;
+ msi_addr.shv = 0;
+ msi_addr.interrupt_format = 1;
+ msi_addr.handle_0_14 = index & 0x7fff;
+ msi_addr.head = 0xfee;
+ msi_data.subhandle = 0;
+
+ return pci_setup_msi(dev, *(uint64_t *)&msi_addr,
+ *(uint32_t *)&msi_data);
+}
+
void vtd_init(void)
{
setup_vm();
diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
index 806d9fa..0725f59 100644
--- a/lib/x86/intel-iommu.h
+++ b/lib/x86/intel-iommu.h
@@ -137,5 +137,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
void vtd_init(void);
void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
+bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id);
#endif
diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
index 755ccb1..cb9a991 100644
--- a/x86/intel-iommu.c
+++ b/x86/intel-iommu.c
@@ -12,8 +12,10 @@
#include "intel-iommu.h"
#include "pci-edu.h"
+#include "x86/apic.h"
#define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
+#define VTD_TEST_IR_MSI ("IR MSI")
void vtd_test_dmar(struct pci_edu_dev *dev)
{
@@ -50,6 +52,43 @@ void vtd_test_dmar(struct pci_edu_dev *dev)
free_page(page);
}
+static volatile bool edu_intr_recved;
+
+static void edu_isr(isr_regs_t *regs)
+{
+ edu_intr_recved = true;
+ eoi();
+}
+
+static void vtd_test_ir(struct pci_edu_dev *dev)
+{
+#define VTD_TEST_VECTOR (0xee)
+ /*
+ * Setup EDU PCI device MSI, using interrupt remapping. By
+ * default, EDU device is using INTx.
+ */
+ if (!vtd_setup_msi(&dev->pci_dev, VTD_TEST_VECTOR, 0)) {
+ printf("edu device does not support MSI, skip test\n");
+ report_skip(VTD_TEST_IR_MSI);
+ return;
+ }
+
+ handle_irq(VTD_TEST_VECTOR, edu_isr);
+ irq_enable();
+
+ /* Manually trigger INTR */
+ edu_reg_writel(dev, EDU_REG_INTR_RAISE, 1);
+
+ while (!edu_intr_recved)
+ cpu_relax();
+
+ /* Clear INTR bits */
+ edu_reg_writel(dev, EDU_REG_INTR_RAISE, 0);
+
+ /* We are good as long as we reach here */
+ report(VTD_TEST_IR_MSI, true);
+}
+
int main(int argc, char *argv[])
{
struct pci_edu_dev dev;
@@ -70,8 +109,11 @@ int main(int argc, char *argv[])
printf("Please specify \"-device edu\" to do "
"further IOMMU tests.\n");
report_skip(VTD_TEST_DMAR_4B);
- } else
+ report_skip(VTD_TEST_IR_MSI);
+ } else {
vtd_test_dmar(&dev);
+ vtd_test_ir(&dev);
+ }
return report_summary();
}
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH kvm-unit-tests v2 17/17] x86/unittests: add intel-iommu test
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (15 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 16/17] x86: intel-iommu: add IR MSI test Peter Xu
@ 2016-11-09 15:10 ` Peter Xu
2016-11-10 20:21 ` Andrew Jones
2016-11-09 15:19 ` [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
2016-11-10 20:25 ` Andrew Jones
18 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:10 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, peterx, agordeev, jan.kiszka, pbonzini
Signed-off-by: Peter Xu <peterx@redhat.com>
---
x86/unittests.cfg | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 23395c6..5413838 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -217,3 +217,10 @@ extra_params = -cpu kvm64,hv_time,hv_synic,hv_stimer -device hyperv-testdev
file = hyperv_clock.flat
smp = 2
extra_params = -cpu kvm64,hv_time
+
+[intel_iommu]
+file = intel-iommu.flat
+arch = x86_64
+timeout = 30
+smp = 4
+extra_params = -M q35,kernel-irqchip=split -device intel-iommu,intremap=on,eim=off -device edu
--
2.7.4
^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 00/17] VT-d unit test
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (16 preceding siblings ...)
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 17/17] x86/unittests: add intel-iommu test Peter Xu
@ 2016-11-09 15:19 ` Peter Xu
2016-11-10 20:25 ` Andrew Jones
18 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-09 15:19 UTC (permalink / raw)
To: kvm; +Cc: drjones, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:07AM -0500, Peter Xu wrote:
> This is v2 of vt-d unit test series.
>
> Patch "libcflat: add IS_ALIGNED() macro, and page sizes" is picked up
> by Drew in the ARM GIC framework series, so please feel free to drop
> it when needed.
Forget to mention: Series v2 is rebased to Alex's PCI v11 series.
Online repo for better reference:
https://github.com/xzpeter/kvm-unit-tests.git iommu-ut-v2
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 02/17] libcflat: introduce is_power_of_2()
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 02/17] libcflat: introduce is_power_of_2() Peter Xu
@ 2016-11-10 18:20 ` Andrew Jones
2016-11-14 20:14 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 18:20 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:09AM -0500, Peter Xu wrote:
> Suggested-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/libcflat.h | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/lib/libcflat.h b/lib/libcflat.h
> index 72b1bf9..19bd0c6 100644
> --- a/lib/libcflat.h
> +++ b/lib/libcflat.h
> @@ -103,4 +103,9 @@ do { \
> } \
> } while (0)
>
> +static inline bool is_power_of_2(unsigned long n)
> +{
> + return (n && !(n & (n - 1)));
nit: outer () are unnecessary
> +}
> +
> #endif
Reviewed-by: Andrew Jones <drjones@redhat.com>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 03/17] x86: intel-iommu: add vt-d init test
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 03/17] x86: intel-iommu: add vt-d init test Peter Xu
@ 2016-11-10 19:09 ` Andrew Jones
2016-11-14 20:18 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:09 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:10AM -0500, Peter Xu wrote:
> Adding fundamental init test for Intel IOMMU. This includes basic
> initialization of Intel IOMMU device, like DMAR (DMA Remapping),
> IR (Interrupt Remapping), QI (Queue Invalidation), etc.
>
> Further tests can use vtd_init() to initialize Intel IOMMU environment.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/x86/intel-iommu.c | 88 +++++++++++++++++++++++++++++++++++++
> lib/x86/intel-iommu.h | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++
> x86/Makefile.x86_64 | 2 +
> x86/intel-iommu.c | 27 ++++++++++++
> 4 files changed, 235 insertions(+)
> create mode 100644 lib/x86/intel-iommu.c
> create mode 100644 lib/x86/intel-iommu.h
> create mode 100644 x86/intel-iommu.c
>
> diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
> new file mode 100644
> index 0000000..6f52697
> --- /dev/null
> +++ b/lib/x86/intel-iommu.c
> @@ -0,0 +1,88 @@
> +/*
> + * Intel IOMMU APIs
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + */
> +
> +#include "intel-iommu.h"
> +
> +#define VTD_RTA_MASK (PAGE_MASK)
> +#define VTD_IRTA_MASK (PAGE_MASK)
> +
> +static uint64_t vtd_root_table(void)
> +{
> + /* No extend root table support yet */
> + return vtd_readq(DMAR_RTADDR_REG) & VTD_RTA_MASK;
Should use tabs. You can run the kernel's scripts/checkpatch.pl,
since we're using the same coding standard (pretty much...)
> +}
> +
> +static uint64_t vtd_ir_table(void)
> +{
> + return vtd_readq(DMAR_IRTA_REG) & VTD_IRTA_MASK;
> +}
The above two functions are both only used once each. Do we need them?
> +
> +static void vtd_gcmd_or(uint32_t cmd)
> +{
> + uint32_t status;
> +
> + /* We only allow set one bit for each time */
> + assert(is_power_of_2(cmd));
> +
> + status = vtd_readl(DMAR_GSTS_REG);
> + vtd_writel(DMAR_GCMD_REG, status | cmd);
> +
> + if (cmd & VTD_GCMD_ONE_SHOT_BITS) {
> + /* One-shot bits are taking effect immediately */
> + return;
> + }
> +
> + /* Make sure IOMMU handled our command request */
> + while (!(vtd_readl(DMAR_GSTS_REG) & cmd))
> + cpu_relax();
> +}
> +
> +static void vtd_dump_init_info(void)
> +{
> + printf("VT-d version: 0x%x\n", vtd_readl(DMAR_VER_REG));
> + printf(" cap: 0x%016lx\n", vtd_readq(DMAR_CAP_REG));
> + printf(" ecap: 0x%016lx\n", vtd_readq(DMAR_ECAP_REG));
> +}
> +
> +static void vtd_setup_root_table(void)
> +{
> + void *root = alloc_page();
> +
> + memset(root, 0, PAGE_SIZE);
> + vtd_writeq(DMAR_RTADDR_REG, virt_to_phys(root));
> + vtd_gcmd_or(VTD_GCMD_ROOT);
> + printf("DMAR table address: 0x%016lx\n", vtd_root_table());
> +}
> +
> +static void vtd_setup_ir_table(void)
> +{
> + void *root = alloc_page();
> +
> + memset(root, 0, PAGE_SIZE);
> + /* 0xf stands for table size (2^(0xf+1) == 65536) */
> + vtd_writeq(DMAR_IRTA_REG, virt_to_phys(root) | 0xf);
> + vtd_gcmd_or(VTD_GCMD_IR_TABLE);
> + printf("IR table address: 0x%016lx\n", vtd_ir_table());
> +}
> +
> +void vtd_init(void)
> +{
> + setup_vm();
> + smp_init();
> +
> + vtd_dump_init_info();
> + vtd_gcmd_or(VTD_GCMD_QI); /* Enable QI */
> + vtd_setup_root_table();
> + vtd_setup_ir_table();
> + vtd_gcmd_or(VTD_GCMD_DMAR); /* Enable DMAR */
> + vtd_gcmd_or(VTD_GCMD_IR); /* Enable IR */
> +}
> diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
> new file mode 100644
> index 0000000..d95d76c
> --- /dev/null
> +++ b/lib/x86/intel-iommu.h
> @@ -0,0 +1,118 @@
> +/*
> + * Intel IOMMU header
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + *
> + * (From include/linux/intel-iommu.h)
> + */
> +
> +#ifndef __INTEL_IOMMU_H__
> +#define __INTEL_IOMMU_H__
> +
> +#include "libcflat.h"
> +#include "vm.h"
> +#include "isr.h"
> +#include "smp.h"
> +#include "desc.h"
> +#include "asm/io.h"
> +
> +#define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
> +
> +/*
> + * Intel IOMMU register specification
> + */
> +#define DMAR_VER_REG 0x0 /* Arch version supported by this IOMMU */
> +#define DMAR_CAP_REG 0x8 /* Hardware supported capabilities */
> +#define DMAR_CAP_REG_HI 0xc /* High 32-bit of DMAR_CAP_REG */
> +#define DMAR_ECAP_REG 0x10 /* Extended capabilities supported */
> +#define DMAR_ECAP_REG_HI 0X14
> +#define DMAR_GCMD_REG 0x18 /* Global command */
> +#define DMAR_GSTS_REG 0x1c /* Global status */
> +#define DMAR_RTADDR_REG 0x20 /* Root entry table */
> +#define DMAR_RTADDR_REG_HI 0X24
> +#define DMAR_CCMD_REG 0x28 /* Context command */
> +#define DMAR_CCMD_REG_HI 0x2c
> +#define DMAR_FSTS_REG 0x34 /* Fault status */
> +#define DMAR_FECTL_REG 0x38 /* Fault control */
> +#define DMAR_FEDATA_REG 0x3c /* Fault event interrupt data */
> +#define DMAR_FEADDR_REG 0x40 /* Fault event interrupt addr */
> +#define DMAR_FEUADDR_REG 0x44 /* Upper address */
> +#define DMAR_AFLOG_REG 0x58 /* Advanced fault control */
> +#define DMAR_AFLOG_REG_HI 0X5c
> +#define DMAR_PMEN_REG 0x64 /* Enable protected memory region */
> +#define DMAR_PLMBASE_REG 0x68 /* PMRR low addr */
> +#define DMAR_PLMLIMIT_REG 0x6c /* PMRR low limit */
> +#define DMAR_PHMBASE_REG 0x70 /* PMRR high base addr */
> +#define DMAR_PHMBASE_REG_HI 0X74
> +#define DMAR_PHMLIMIT_REG 0x78 /* PMRR high limit */
> +#define DMAR_PHMLIMIT_REG_HI 0x7c
> +#define DMAR_IQH_REG 0x80 /* Invalidation queue head */
> +#define DMAR_IQH_REG_HI 0X84
> +#define DMAR_IQT_REG 0x88 /* Invalidation queue tail */
> +#define DMAR_IQT_REG_HI 0X8c
> +#define DMAR_IQA_REG 0x90 /* Invalidation queue addr */
> +#define DMAR_IQA_REG_HI 0x94
> +#define DMAR_ICS_REG 0x9c /* Invalidation complete status */
> +#define DMAR_IRTA_REG 0xb8 /* Interrupt remapping table addr */
> +#define DMAR_IRTA_REG_HI 0xbc
> +#define DMAR_IECTL_REG 0xa0 /* Invalidation event control */
> +#define DMAR_IEDATA_REG 0xa4 /* Invalidation event data */
> +#define DMAR_IEADDR_REG 0xa8 /* Invalidation event address */
> +#define DMAR_IEUADDR_REG 0xac /* Invalidation event address */
> +#define DMAR_PQH_REG 0xc0 /* Page request queue head */
> +#define DMAR_PQH_REG_HI 0xc4
> +#define DMAR_PQT_REG 0xc8 /* Page request queue tail*/
> +#define DMAR_PQT_REG_HI 0xcc
> +#define DMAR_PQA_REG 0xd0 /* Page request queue address */
> +#define DMAR_PQA_REG_HI 0xd4
> +#define DMAR_PRS_REG 0xdc /* Page request status */
> +#define DMAR_PECTL_REG 0xe0 /* Page request event control */
> +#define DMAR_PEDATA_REG 0xe4 /* Page request event data */
> +#define DMAR_PEADDR_REG 0xe8 /* Page request event address */
> +#define DMAR_PEUADDR_REG 0xec /* Page event upper address */
> +#define DMAR_MTRRCAP_REG 0x100 /* MTRR capability */
> +#define DMAR_MTRRCAP_REG_HI 0x104
> +#define DMAR_MTRRDEF_REG 0x108 /* MTRR default type */
> +#define DMAR_MTRRDEF_REG_HI 0x10c
> +
> +#define VTD_GCMD_IR_TABLE (0x1000000)
> +#define VTD_GCMD_IR (0x2000000)
> +#define VTD_GCMD_QI (0x4000000)
> +#define VTD_GCMD_WBF (0x8000000) /* Write Buffer Flush */
> +#define VTD_GCMD_SFL (0x20000000) /* Set Fault Log */
> +#define VTD_GCMD_ROOT (0x40000000)
> +#define VTD_GCMD_DMAR (0x80000000)
> +#define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
> + VTD_GCMD_SFL | VTD_GCMD_ROOT)
> +
> +#define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
> +
> +static inline void vtd_writel(unsigned int reg, uint32_t value)
> +{
> + __raw_writel(value, vtd_reg(reg));
> +}
> +
> +static inline void vtd_writeq(unsigned int reg, uint64_t value)
> +{
> + __raw_writeq(value, vtd_reg(reg));
> +}
> +
> +static inline uint32_t vtd_readl(unsigned int reg)
> +{
> + return __raw_readl(vtd_reg(reg));
> +}
> +
> +static inline uint64_t vtd_readq(unsigned int reg)
> +{
> + return __raw_readq(vtd_reg(reg));
> +}
> +
> +void vtd_init(void);
> +
> +#endif
> diff --git a/x86/Makefile.x86_64 b/x86/Makefile.x86_64
> index f82492b..3e2821e 100644
> --- a/x86/Makefile.x86_64
> +++ b/x86/Makefile.x86_64
> @@ -4,6 +4,7 @@ ldarch = elf64-x86-64
> CFLAGS += -mno-red-zone
>
> cflatobjs += lib/x86/setjmp64.o
> +cflatobjs += lib/x86/intel-iommu.o
>
> tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
> $(TEST_DIR)/emulator.flat $(TEST_DIR)/idt_test.flat \
> @@ -14,6 +15,7 @@ tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
> tests += $(TEST_DIR)/svm.flat
> tests += $(TEST_DIR)/vmx.flat
> tests += $(TEST_DIR)/tscdeadline_latency.flat
> +tests += $(TEST_DIR)/intel-iommu.flat
>
> include $(TEST_DIR)/Makefile.common
>
> diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
> new file mode 100644
> index 0000000..f247913
> --- /dev/null
> +++ b/x86/intel-iommu.c
> @@ -0,0 +1,27 @@
> +/*
> + * Intel IOMMU unit test.
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + */
> +
> +#include "intel-iommu.h"
> +
> +int main(int argc, char *argv[])
> +{
> + vtd_init();
> +
> + report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
> + report("QI enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_QI);
> + report("DMAR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_ROOT);
> + report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
> + report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
> + report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
> +
> + return report_summary();
> +}
> --
> 2.7.4
>
Looks good to me, but I only reviewed it wrt framework/style.
drew
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 07/17] pci: introduce struct pci_dev
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 07/17] pci: introduce struct pci_dev Peter Xu
@ 2016-11-10 19:21 ` Andrew Jones
2016-11-14 20:22 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:21 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:14AM -0500, Peter Xu wrote:
> To extend current PCI framework, we need a per-device struct to store
> device specific information. Time to have a pci_dev struct. Most of the
> current PCI APIs are converted to use this pci_dev object as the first
> argument. Currently it only contains one field "bdf", which is the bdf
> of current device.
>
> For a few APIs like pci_config_*() ops or pci_find_dev(), I kept the old
> interface (use PCI BDF value rather than "struct pci_dev") since they
> can be used in a open context that without any specific PCI device.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci-host-generic.c | 9 +++++---
> lib/pci-testdev.c | 10 +++++----
> lib/pci.c | 57 ++++++++++++++++++++++++++++++--------------------
> lib/pci.h | 24 ++++++++++++++-------
> x86/vmexit.c | 18 +++++++++-------
> 5 files changed, 72 insertions(+), 46 deletions(-)
>
> diff --git a/lib/pci-host-generic.c b/lib/pci-host-generic.c
> index 6ac0f15..ea83d72 100644
> --- a/lib/pci-host-generic.c
> +++ b/lib/pci-host-generic.c
> @@ -211,6 +211,7 @@ static bool pci_alloc_resource(pcidevaddr_t dev, int bar_num, u64 *addr)
>
> bool pci_probe(void)
> {
> + struct pci_dev pci_dev;
> pcidevaddr_t dev;
> u8 header;
> u32 cmd;
> @@ -225,6 +226,8 @@ bool pci_probe(void)
> if (!pci_dev_exists(dev))
> continue;
>
> + pci_dev_init(&pci_dev, dev);
> +
> /* We are only interested in normal PCI devices */
> header = pci_config_readb(dev, PCI_HEADER_TYPE);
> if ((header & PCI_HEADER_TYPE_MASK) != PCI_HEADER_TYPE_NORMAL)
> @@ -236,15 +239,15 @@ bool pci_probe(void)
> u64 addr;
>
> if (pci_alloc_resource(dev, i, &addr)) {
> - pci_bar_set_addr(dev, i, addr);
> + pci_bar_set_addr(&pci_dev, i, addr);
>
> - if (pci_bar_is_memory(dev, i))
> + if (pci_bar_is_memory(&pci_dev, i))
> cmd |= PCI_COMMAND_MEMORY;
> else
> cmd |= PCI_COMMAND_IO;
> }
>
> - if (pci_bar_is64(dev, i))
> + if (pci_bar_is64(&pci_dev, i))
> i++;
> }
>
> diff --git a/lib/pci-testdev.c b/lib/pci-testdev.c
> index ad482d3..7d298e6 100644
> --- a/lib/pci-testdev.c
> +++ b/lib/pci-testdev.c
> @@ -163,9 +163,10 @@ static int pci_testdev_all(struct pci_test_dev_hdr *test,
>
> int pci_testdev(void)
> {
> + struct pci_dev pci_dev;
> + pcidevaddr_t dev;
> phys_addr_t addr;
> void __iomem *mem, *io;
> - pcidevaddr_t dev;
> int nr_tests = 0;
> bool ret;
>
> @@ -175,14 +176,15 @@ int pci_testdev(void)
> "check QEMU '-device pci-testdev' parameter\n");
> return -1;
> }
> + pci_dev_init(&pci_dev, dev);
>
> - ret = pci_bar_is_valid(dev, 0) && pci_bar_is_valid(dev, 1);
> + ret = pci_bar_is_valid(&pci_dev, 0) && pci_bar_is_valid(&pci_dev, 1);
> assert(ret);
>
> - addr = pci_bar_get_addr(dev, 0);
> + addr = pci_bar_get_addr(&pci_dev, 0);
> mem = ioremap(addr, PAGE_SIZE);
>
> - addr = pci_bar_get_addr(dev, 1);
> + addr = pci_bar_get_addr(&pci_dev, 1);
> io = (void *)(unsigned long)addr;
>
> nr_tests += pci_testdev_all(mem, &pci_testdev_mem_ops);
> diff --git a/lib/pci.c b/lib/pci.c
> index 6bd54cb..c0bbcba 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -13,16 +13,21 @@ bool pci_dev_exists(pcidevaddr_t dev)
> pci_config_readw(dev, PCI_DEVICE_ID) != 0xffff);
> }
>
> +void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf)
> +{
> + memset(dev, 0, sizeof(*dev));
> + dev->bdf = bdf;
> +}
> +
> /* Scan bus look for a specific device. Only bus 0 scanned for now. */
> pcidevaddr_t pci_find_dev(uint16_t vendor_id, uint16_t device_id)
> {
> pcidevaddr_t dev;
>
> - for (dev = 0; dev < 256; ++dev) {
> + for (dev = 0; dev < PCI_DEVFN_MAX; ++dev)
> if (pci_config_readw(dev, PCI_VENDOR_ID) == vendor_id &&
> pci_config_readw(dev, PCI_DEVICE_ID) == device_id)
> return dev;
> - }
I liked the {} here because the "one" line spans three.
>
> return PCIDEVADDR_INVALID;
> }
> @@ -33,12 +38,13 @@ uint32_t pci_bar_mask(uint32_t bar)
> PCI_BASE_ADDRESS_IO_MASK : PCI_BASE_ADDRESS_MEM_MASK;
> }
>
> -uint32_t pci_bar_get(pcidevaddr_t dev, int bar_num)
> +uint32_t pci_bar_get(struct pci_dev *dev, int bar_num)
> {
> - return pci_config_readl(dev, PCI_BASE_ADDRESS_0 + bar_num * 4);
> + return pci_config_readl(dev->bdf, PCI_BASE_ADDRESS_0 +
> + bar_num * 4);
> }
>
> -phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num)
> +phys_addr_t pci_bar_get_addr(struct pci_dev *dev, int bar_num)
> {
> uint32_t bar = pci_bar_get(dev, bar_num);
> uint32_t mask = pci_bar_mask(bar);
> @@ -47,17 +53,18 @@ phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num)
> if (pci_bar_is64(dev, bar_num))
> addr |= (uint64_t)pci_bar_get(dev, bar_num + 1) << 32;
>
> - return pci_translate_addr(dev, addr);
> + return pci_translate_addr(dev->bdf, addr);
> }
>
> -void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr)
> +void pci_bar_set_addr(struct pci_dev *dev, int bar_num, phys_addr_t addr)
> {
> int off = PCI_BASE_ADDRESS_0 + bar_num * 4;
>
> - pci_config_writel(dev, off, (uint32_t)addr);
> + pci_config_writel(dev->bdf, off, (uint32_t)addr);
>
> if (pci_bar_is64(dev, bar_num))
> - pci_config_writel(dev, off + 4, (uint32_t)(addr >> 32));
> + pci_config_writel(dev->bdf, off + 4,
> + (uint32_t)(addr >> 32));
> }
>
> /*
> @@ -70,20 +77,21 @@ void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr)
> * The following pci_bar_size_helper() and pci_bar_size() functions
> * implement the algorithm.
> */
> -static uint32_t pci_bar_size_helper(pcidevaddr_t dev, int bar_num)
> +static uint32_t pci_bar_size_helper(struct pci_dev *dev, int bar_num)
> {
> int off = PCI_BASE_ADDRESS_0 + bar_num * 4;
> + uint16_t bdf = dev->bdf;
> uint32_t bar, val;
>
> - bar = pci_config_readl(dev, off);
> - pci_config_writel(dev, off, ~0u);
> - val = pci_config_readl(dev, off);
> - pci_config_writel(dev, off, bar);
> + bar = pci_config_readl(bdf, off);
> + pci_config_writel(bdf, off, ~0u);
> + val = pci_config_readl(bdf, off);
> + pci_config_writel(bdf, off, bar);
>
> return val;
> }
>
> -phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num)
> +phys_addr_t pci_bar_size(struct pci_dev *dev, int bar_num)
> {
> uint32_t bar, size;
>
> @@ -104,19 +112,19 @@ phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num)
> }
> }
>
> -bool pci_bar_is_memory(pcidevaddr_t dev, int bar_num)
> +bool pci_bar_is_memory(struct pci_dev *dev, int bar_num)
> {
> uint32_t bar = pci_bar_get(dev, bar_num);
>
> return !(bar & PCI_BASE_ADDRESS_SPACE_IO);
> }
>
> -bool pci_bar_is_valid(pcidevaddr_t dev, int bar_num)
> +bool pci_bar_is_valid(struct pci_dev *dev, int bar_num)
> {
> return pci_bar_get(dev, bar_num);
> }
>
> -bool pci_bar_is64(pcidevaddr_t dev, int bar_num)
> +bool pci_bar_is64(struct pci_dev *dev, int bar_num)
> {
> uint32_t bar = pci_bar_get(dev, bar_num);
>
> @@ -127,7 +135,7 @@ bool pci_bar_is64(pcidevaddr_t dev, int bar_num)
> PCI_BASE_ADDRESS_MEM_TYPE_64;
> }
>
> -void pci_bar_print(pcidevaddr_t dev, int bar_num)
> +void pci_bar_print(struct pci_dev *dev, int bar_num)
> {
> phys_addr_t size, start, end;
> uint32_t bar;
> @@ -187,6 +195,9 @@ static void pci_dev_print(pcidevaddr_t dev)
> uint8_t subclass = pci_config_readb(dev, PCI_CLASS_DEVICE);
> uint8_t class = pci_config_readb(dev, PCI_CLASS_DEVICE + 1);
> int i;
> + struct pci_dev pci_dev;
Putting that above the 'int i' would appease my aesthetics OCD...
> +
> + pci_dev_init(&pci_dev, dev);
>
> pci_dev_print_id(dev);
> printf(" type %02x progif %02x class %02x subclass %02x\n",
> @@ -196,12 +207,12 @@ static void pci_dev_print(pcidevaddr_t dev)
> return;
>
> for (i = 0; i < 6; i++) {
> - if (pci_bar_size(dev, i)) {
> + if (pci_bar_size(&pci_dev, i)) {
> printf("\t");
> - pci_bar_print(dev, i);
> + pci_bar_print(&pci_dev, i);
> printf("\n");
> }
> - if (pci_bar_is64(dev, i))
> + if (pci_bar_is64(&pci_dev, i))
> i++;
> }
> }
> @@ -210,7 +221,7 @@ void pci_print(void)
> {
> pcidevaddr_t dev;
>
> - for (dev = 0; dev < 256; ++dev) {
> + for (dev = 0; dev < PCI_DEVFN_MAX; ++dev) {
> if (pci_dev_exists(dev))
> pci_dev_print(dev);
> }
> diff --git a/lib/pci.h b/lib/pci.h
> index 30f5381..21f5a7b 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -15,6 +15,14 @@ enum {
> PCIDEVADDR_INVALID = 0xffff,
> };
>
> +#define PCI_DEVFN_MAX (256)
> +
> +struct pci_dev {
> + uint16_t bdf;
> +};
> +
> +void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> +
> extern bool pci_probe(void);
> extern void pci_print(void);
> extern bool pci_dev_exists(pcidevaddr_t dev);
> @@ -32,15 +40,15 @@ extern pcidevaddr_t pci_find_dev(uint16_t vendor_id, uint16_t device_id);
> * It is expected the caller is aware of the device BAR layout and never
> * tries to address the middle of a 64-bit register.
> */
> -extern phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num);
> -extern void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr);
> -extern phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num);
> -extern uint32_t pci_bar_get(pcidevaddr_t dev, int bar_num);
> +extern phys_addr_t pci_bar_get_addr(struct pci_dev *dev, int bar_num);
> +extern void pci_bar_set_addr(struct pci_dev *dev, int bar_num, phys_addr_t addr);
> +extern phys_addr_t pci_bar_size(struct pci_dev *dev, int bar_num);
> +extern uint32_t pci_bar_get(struct pci_dev *dev, int bar_num);
> extern uint32_t pci_bar_mask(uint32_t bar);
> -extern bool pci_bar_is64(pcidevaddr_t dev, int bar_num);
> -extern bool pci_bar_is_memory(pcidevaddr_t dev, int bar_num);
> -extern bool pci_bar_is_valid(pcidevaddr_t dev, int bar_num);
> -extern void pci_bar_print(pcidevaddr_t dev, int bar_num);
> +extern bool pci_bar_is64(struct pci_dev *dev, int bar_num);
> +extern bool pci_bar_is_memory(struct pci_dev *dev, int bar_num);
> +extern bool pci_bar_is_valid(struct pci_dev *dev, int bar_num);
> +extern void pci_bar_print(struct pci_dev *dev, int bar_num);
> extern void pci_dev_print_id(pcidevaddr_t dev);
>
> int pci_testdev(void);
> diff --git a/x86/vmexit.c b/x86/vmexit.c
> index 2d99d5f..63fa070 100644
> --- a/x86/vmexit.c
> +++ b/x86/vmexit.c
> @@ -372,7 +372,8 @@ int main(int ac, char **av)
> struct fadt_descriptor_rev1 *fadt;
> int i;
> unsigned long membar = 0;
> - pcidevaddr_t pcidev;
> + struct pci_dev pcidev;
> + int ret;
>
> smp_init();
> setup_vm();
> @@ -385,21 +386,22 @@ int main(int ac, char **av)
> pm_tmr_blk = fadt->pm_tmr_blk;
> printf("PM timer port is %x\n", pm_tmr_blk);
>
> - pcidev = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
> - if (pcidev != PCIDEVADDR_INVALID) {
> + ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
> + if (ret != PCIDEVADDR_INVALID) {
> + pci_dev_init(&pcidev, ret);
> for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
> - if (!pci_bar_is_valid(pcidev, i)) {
> + if (!pci_bar_is_valid(&pcidev, i)) {
> continue;
> }
> - if (pci_bar_is_memory(pcidev, i)) {
> - membar = pci_bar_get_addr(pcidev, i);
> + if (pci_bar_is_memory(&pcidev, i)) {
> + membar = pci_bar_get_addr(&pcidev, i);
> pci_test.memaddr = ioremap(membar, PAGE_SIZE);
> } else {
> - pci_test.iobar = pci_bar_get_addr(pcidev, i);
> + pci_test.iobar = pci_bar_get_addr(&pcidev, i);
> }
> }
> printf("pci-testdev at 0x%x membar %lx iobar %x\n",
> - pcidev, membar, pci_test.iobar);
> + pcidev.bdf, membar, pci_test.iobar);
> }
>
> for (i = 0; i < ARRAY_SIZE(tests); ++i)
> --
> 2.7.4
>
Besides my nits
Reviewed-by: Andrew Jones <drjones@redhat.com>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars()
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars() Peter Xu
@ 2016-11-10 19:24 ` Andrew Jones
2016-11-14 20:33 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:24 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:15AM -0500, Peter Xu wrote:
> Let's provide a more general way to scan PCI bars, rather than read the
> config registers every time.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.c | 13 ++++++++++++-
> lib/pci.h | 3 +++
> 2 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/lib/pci.c b/lib/pci.c
> index c0bbcba..c063d53 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -206,7 +206,7 @@ static void pci_dev_print(pcidevaddr_t dev)
> if ((header & PCI_HEADER_TYPE_MASK) != PCI_HEADER_TYPE_NORMAL)
> return;
>
> - for (i = 0; i < 6; i++) {
> + for (i = 0; i < PCI_BAR_NUM; i++) {
> if (pci_bar_size(&pci_dev, i)) {
> printf("\t");
> pci_bar_print(&pci_dev, i);
> @@ -226,3 +226,14 @@ void pci_print(void)
> pci_dev_print(dev);
> }
> }
> +
> +void pci_scan_bars(struct pci_dev *dev)
> +{
> + int i = 0;
> +
> + for (i = 0; i < PCI_BAR_NUM; i++) {
> + if (!pci_bar_is_valid(dev, i))
> + continue;
> + dev->bar[i] = pci_bar_get_addr(dev, i);
What happens when you get_addr a 64-bit bar in the middle?
Shouldn't we skip that?
> + }
> +}
> diff --git a/lib/pci.h b/lib/pci.h
> index 21f5a7b..5ed4e11 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -15,13 +15,16 @@ enum {
> PCIDEVADDR_INVALID = 0xffff,
> };
>
> +#define PCI_BAR_NUM (6)
> #define PCI_DEVFN_MAX (256)
>
> struct pci_dev {
> uint16_t bdf;
> + phys_addr_t bar[PCI_BAR_NUM];
> };
>
> void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> +void pci_scan_bars(struct pci_dev *dev);
>
> extern bool pci_probe(void);
> extern void pci_print(void);
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 09/17] x86/vmexit: leverage pci_scan_bars()
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 09/17] x86/vmexit: leverage pci_scan_bars() Peter Xu
@ 2016-11-10 19:27 ` Andrew Jones
2016-11-14 20:35 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:27 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:16AM -0500, Peter Xu wrote:
> Since pci-testdev is a very specific device for QEMU, let's use the new
> pci_scan_bars() helper, and selectively choose the bars we want.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.h | 2 ++
> x86/vmexit.c | 17 ++++++-----------
> 2 files changed, 8 insertions(+), 11 deletions(-)
>
> diff --git a/lib/pci.h b/lib/pci.h
> index 5ed4e11..e452819 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -68,6 +68,8 @@ int pci_testdev(void);
> * pci-testdev supports at least three types of tests (via mmio and
> * portio BARs): no-eventfd, wildcard-eventfd and datamatch-eventfd
> */
> +#define PCI_TESTDEV_BAR_MEM 0
> +#define PCI_TESTDEV_BAR_IO 1
> #define PCI_TESTDEV_NUM_BARS 2
> #define PCI_TESTDEV_NUM_TESTS 3
>
> diff --git a/x86/vmexit.c b/x86/vmexit.c
> index 63fa070..a22f43f 100644
> --- a/x86/vmexit.c
> +++ b/x86/vmexit.c
> @@ -389,17 +389,12 @@ int main(int ac, char **av)
> ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
> if (ret != PCIDEVADDR_INVALID) {
> pci_dev_init(&pcidev, ret);
> - for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
> - if (!pci_bar_is_valid(&pcidev, i)) {
> - continue;
> - }
> - if (pci_bar_is_memory(&pcidev, i)) {
> - membar = pci_bar_get_addr(&pcidev, i);
> - pci_test.memaddr = ioremap(membar, PAGE_SIZE);
> - } else {
> - pci_test.iobar = pci_bar_get_addr(&pcidev, i);
> - }
> - }
> + pci_scan_bars(&pcidev);
> + assert(pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_MEM));
> + assert(!pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_IO));
> + membar = pcidev.bar[PCI_TESTDEV_BAR_MEM];
nit: I'd drop 'membar' and just pass pcidev.bar to ioremap
> + pci_test.memaddr = ioremap(membar, PAGE_SIZE);
> + pci_test.iobar = pcidev.bar[PCI_TESTDEV_BAR_IO];
> printf("pci-testdev at 0x%x membar %lx iobar %x\n",
> pcidev.bdf, membar, pci_test.iobar);
> }
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 10/17] pci: provide pci_cmd_set_clr()
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 10/17] pci: provide pci_cmd_set_clr() Peter Xu
@ 2016-11-10 19:31 ` Andrew Jones
0 siblings, 0 replies; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:31 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:17AM -0500, Peter Xu wrote:
> Helper function to set/clear specific bit in PCI_COMMAND register.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.c | 12 ++++++++++++
> lib/pci.h | 1 +
> 2 files changed, 13 insertions(+)
>
> diff --git a/lib/pci.c b/lib/pci.c
> index c063d53..fd17ea5 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -7,6 +7,18 @@
> #include "pci.h"
> #include "asm/pci.h"
>
> +void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
> +{
> + uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
> +
> + /* No overlap is allowed */
> + assert((set & clr) == 0);
> + val |= set;
> + val &= ~clr;
> +
> + pci_config_writew(dev->bdf, PCI_COMMAND, val);
> +}
> +
> bool pci_dev_exists(pcidevaddr_t dev)
> {
> return (pci_config_readw(dev, PCI_VENDOR_ID) != 0xffff &&
> diff --git a/lib/pci.h b/lib/pci.h
> index e452819..de15086 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -25,6 +25,7 @@ struct pci_dev {
>
> void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> void pci_scan_bars(struct pci_dev *dev);
> +void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
>
> extern bool pci_probe(void);
> extern void pci_print(void);
> --
> 2.7.4
>
Reviewed-by: Andrew Jones <drjones@redhat.com>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 11/17] pci: provide pci_enable_defaults()
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 11/17] pci: provide pci_enable_defaults() Peter Xu
@ 2016-11-10 19:33 ` Andrew Jones
2016-11-14 20:42 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:33 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:18AM -0500, Peter Xu wrote:
> Provide a function to do most of the common PCI init work.
>
> Suggested-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.c | 8 ++++++++
> lib/pci.h | 1 +
> 2 files changed, 9 insertions(+)
>
> diff --git a/lib/pci.c b/lib/pci.c
> index fd17ea5..971f02e 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -249,3 +249,11 @@ void pci_scan_bars(struct pci_dev *dev)
> dev->bar[i] = pci_bar_get_addr(dev, i);
> }
> }
> +
> +int pci_enable_defaults(struct pci_dev *dev)
> +{
> + pci_scan_bars(dev);
> + /* Enable device DMA operations */
> + pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
> + return 0;
Shouldn't this be a void function that just asserts on
any errors it detects? I'm not sure why we're [currently
unconditionally] returning zero.
> +}
> diff --git a/lib/pci.h b/lib/pci.h
> index de15086..63f8a0a 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -26,6 +26,7 @@ struct pci_dev {
> void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> void pci_scan_bars(struct pci_dev *dev);
> void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> +int pci_enable_defaults(struct pci_dev *dev);
>
> extern bool pci_probe(void);
> extern void pci_print(void);
> --
> 2.7.4
>
Otherwise
Reviewed-by: Andrew Jones <drjones@redhat.com>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 13/17] pci: edu: introduce pci-edu helpers
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 13/17] pci: edu: introduce pci-edu helpers Peter Xu
@ 2016-11-10 19:45 ` Andrew Jones
2016-11-14 20:48 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:45 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:20AM -0500, Peter Xu wrote:
> QEMU edu device is a pci device that is originally written for
> educational purpose, however it also suits for IOMMU unit test. Adding
> helpers for this specific device to implement the device logic.
>
> The device supports lots of functions, here only DMA operation is
> supported.
>
> The spec of the device can be found at:
>
> https://github.com/qemu/qemu/blob/master/docs/specs/edu.txt
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci-edu.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/pci-edu.h | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 155 insertions(+)
> create mode 100644 lib/pci-edu.c
> create mode 100644 lib/pci-edu.h
>
> diff --git a/lib/pci-edu.c b/lib/pci-edu.c
> new file mode 100644
> index 0000000..c9a8b0c
> --- /dev/null
> +++ b/lib/pci-edu.c
> @@ -0,0 +1,73 @@
> +/*
> + * Edu PCI device.
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + */
> +
> +#include "pci-edu.h"
> +#include "asm/barrier.h"
> +
> +/* Return true if alive */
> +static inline bool edu_check_alive(struct pci_edu_dev *dev)
> +{
> + static uint32_t live_count = 1;
> + uint32_t value;
> +
> + edu_reg_writel(dev, EDU_REG_ALIVE, live_count++);
> + value = edu_reg_readl(dev, EDU_REG_ALIVE);
> + return (live_count - 1 == ~value);
> +}
> +
> +bool edu_init(struct pci_edu_dev *dev)
> +{
> + pcidevaddr_t dev_addr;
> +
> + dev_addr = pci_find_dev(PCI_VENDOR_ID_QEMU, PCI_DEVICE_ID_EDU);
> + if (dev_addr == PCIDEVADDR_INVALID)
> + return false;
> +
> + pci_dev_init(&dev->pci_dev, dev_addr);
> + pci_enable_defaults(&dev->pci_dev);
> + assert(edu_check_alive(dev));
> + return true;
> +}
> +
> +void edu_dma(struct pci_edu_dev *dev, iova_t iova,
> + size_t size, int dev_offset, bool from_device)
> +{
> + uint64_t from, to;
> + uint32_t cmd = EDU_CMD_DMA_START;
> +
> + assert(size <= EDU_DMA_SIZE_MAX);
> + assert(dev_offset < EDU_DMA_SIZE_MAX &&
> + dev_offset >= 0);
> +
> + printf("edu device DMA start %s addr %p size 0x%lu off 0x%x\n",
> + from_device ? "FROM" : "TO",
> + (void *)iova, size, dev_offset);
> +
> + if (from_device) {
> + from = dev_offset + EDU_DMA_START;
> + to = iova;
> + cmd |= EDU_CMD_DMA_FROM;
> + } else {
> + from = iova;
> + to = EDU_DMA_START + dev_offset;
> + cmd |= EDU_CMD_DMA_TO;
> + }
> +
> + edu_reg_writeq(dev, EDU_REG_DMA_SRC, from);
> + edu_reg_writeq(dev, EDU_REG_DMA_DST, to);
> + edu_reg_writeq(dev, EDU_REG_DMA_COUNT, size);
> + edu_reg_writel(dev, EDU_REG_DMA_CMD, cmd);
> +
> + /* Wait until DMA finished */
> + while (edu_reg_readl(dev, EDU_REG_DMA_CMD) & EDU_CMD_DMA_START)
> + cpu_relax();
> +}
> diff --git a/lib/pci-edu.h b/lib/pci-edu.h
> new file mode 100644
> index 0000000..af457df
> --- /dev/null
> +++ b/lib/pci-edu.h
> @@ -0,0 +1,82 @@
> +/*
> + * Edu PCI device header.
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + *
> + * Edu device is a virtualized device in QEMU. Please refer to
> + * docs/specs/edu.txt in QEMU repository for EDU device manual.
> + */
> +#ifndef __PCI_EDU_H__
> +#define __PCI_EDU_H__
> +
> +#include "pci.h"
> +#include "asm/io.h"
> +
> +#define PCI_VENDOR_ID_QEMU (0x1234)
> +#define PCI_DEVICE_ID_EDU (0x11e8)
> +
> +/* The only bar used by EDU device */
> +#define EDU_BAR_MEM (0)
> +#define EDU_MAGIC (0xed)
> +#define EDU_VERSION (0x100)
> +#define EDU_DMA_BUF_SIZE (1 << 20)
> +#define EDU_INPUT_BUF_SIZE (256)
> +
> +#define EDU_REG_ID (0x0)
> +#define EDU_REG_ALIVE (0x4)
> +#define EDU_REG_FACTORIAL (0x8)
> +#define EDU_REG_STATUS (0x20)
> +#define EDU_REG_DMA_SRC (0x80)
> +#define EDU_REG_DMA_DST (0x88)
> +#define EDU_REG_DMA_COUNT (0x90)
> +#define EDU_REG_DMA_CMD (0x98)
> +
> +#define EDU_CMD_DMA_START (0x01)
> +#define EDU_CMD_DMA_FROM (0x02)
> +#define EDU_CMD_DMA_TO (0x00)
> +
> +#define EDU_STATUS_FACTORIAL (0x1)
> +#define EDU_STATUS_INT_ENABLE (0x80)
> +
> +#define EDU_DMA_START (0x40000)
> +#define EDU_DMA_SIZE_MAX (4096)
General comment for whole series; I'm not a big fan of the
() around the numeric constants. They're not necessary and
starting to hurt my eyes :-)
> +
> +struct pci_edu_dev {
> + struct pci_dev pci_dev;
> +};
> +
> +#define edu_reg(d, r) (volatile void *)((d)->pci_dev.bar[EDU_BAR_MEM] + (r))
> +
> +static inline uint64_t edu_reg_readq(struct pci_edu_dev *dev, int reg)
> +{
> + return __raw_readq(edu_reg(dev, reg));
> +}
> +
> +static inline uint32_t edu_reg_readl(struct pci_edu_dev *dev, int reg)
> +{
> + return __raw_readl(edu_reg(dev, reg));
> +}
> +
> +static inline void edu_reg_writeq(struct pci_edu_dev *dev, int reg,
> + uint64_t val)
> +{
> + __raw_writeq(val, edu_reg(dev, reg));
> +}
> +
> +static inline void edu_reg_writel(struct pci_edu_dev *dev, int reg,
> + uint32_t val)
> +{
> + __raw_writel(val, edu_reg(dev, reg));
> +}
> +
> +bool edu_init(struct pci_edu_dev *dev);
> +void edu_dma(struct pci_edu_dev *dev, iova_t iova,
> + size_t size, int dev_offset, bool from_device);
> +
> +#endif
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 14/17] x86: intel-iommu: add dmar test
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 14/17] x86: intel-iommu: add dmar test Peter Xu
@ 2016-11-10 19:53 ` Andrew Jones
2016-11-14 20:54 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 19:53 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:21AM -0500, Peter Xu wrote:
> DMAR test is based on QEMU edu device. A 4B DMA memory copy is carried
> out as the simplest DMAR test.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.h | 2 +
> lib/x86/intel-iommu.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/x86/intel-iommu.h | 23 +++++++++
> x86/Makefile.common | 1 +
> x86/intel-iommu.c | 50 +++++++++++++++++++
> 5 files changed, 207 insertions(+)
>
> diff --git a/lib/pci.h b/lib/pci.h
> index 0bdfd8c..c4fef98 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -31,6 +31,8 @@ void pci_scan_bars(struct pci_dev *dev);
> void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> int pci_enable_defaults(struct pci_dev *dev);
>
> +typedef phys_addr_t iova_t;
> +
> extern bool pci_probe(void);
> extern void pci_print(void);
> extern bool pci_dev_exists(pcidevaddr_t dev);
> diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
> index 6f52697..912a379 100644
> --- a/lib/x86/intel-iommu.c
> +++ b/lib/x86/intel-iommu.c
> @@ -11,6 +11,42 @@
> */
>
> #include "intel-iommu.h"
> +#include "libcflat.h"
> +
> +/*
> + * VT-d in QEMU currently only support 39 bits address width, which is
> + * 3-level translation.
> + */
> +#define VTD_PAGE_LEVEL (3)
> +#define VTD_CE_AW_39BIT (0x1)
> +
> +typedef uint64_t vtd_pte_t;
> +
> +struct vtd_root_entry {
> + /* Quad 1 */
> + uint64_t present:1;
> + uint64_t __reserved:11;
> + uint64_t context_table_p:52;
> + /* Quad 2 */
> + uint64_t __reserved_2;
> +} __attribute__ ((packed));
> +typedef struct vtd_root_entry vtd_re_t;
> +
> +struct vtd_context_entry {
> + /* Quad 1 */
> + uint64_t present:1;
> + uint64_t disable_fault_report:1;
> + uint64_t trans_type:2;
> + uint64_t __reserved:8;
> + uint64_t slptptr:52;
> + /* Quad 2 */
> + uint64_t addr_width:3;
> + uint64_t __ignore:4;
> + uint64_t __reserved_2:1;
> + uint64_t domain_id:16;
> + uint64_t __reserved_3:40;
> +} __attribute__ ((packed));
> +typedef struct vtd_context_entry vtd_ce_t;
>
> #define VTD_RTA_MASK (PAGE_MASK)
> #define VTD_IRTA_MASK (PAGE_MASK)
> @@ -74,6 +110,101 @@ static void vtd_setup_ir_table(void)
> printf("IR table address: 0x%016lx\n", vtd_ir_table());
> }
>
> +static void vtd_install_pte(vtd_pte_t *root, iova_t iova,
> + phys_addr_t pa, int level_target)
> +{
> + int level;
> + unsigned int offset;
> + void *page;
> +
> + for (level = VTD_PAGE_LEVEL; level > level_target; level--) {
> + offset = PGDIR_OFFSET(iova, level);
> + if (!(root[offset] & VTD_PTE_RW)) {
> + page = alloc_page();
> + memset(page, 0, PAGE_SIZE);
> + root[offset] = virt_to_phys(page) | VTD_PTE_RW;
> + }
> + root = (uint64_t *)(root[offset] & VTD_PTE_ADDR);
> + }
> +
> + offset = PGDIR_OFFSET(iova, level);
> + root[offset] = pa | VTD_PTE_RW;
> + if (level != 1)
> + /* This is huge page */
> + root[offset] |= VTD_PTE_HUGE;
> +}
> +
> +#define VTD_FETCH_VIRT_ADDR(x) \
> + ((void *)(((uint64_t)phys_to_virt(x)) >> PAGE_SHIFT))
> +
> +/**
> + * vtd_map_range: setup IO address mapping for specific memory range
> + *
> + * @sid: source ID of the device to setup
> + * @iova: start IO virtual address
> + * @pa: start physical address
> + * @size: size of the mapping area
> + */
> +void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
> +{
> + uint8_t bus_n, devfn;
> + void *slptptr;
> + vtd_ce_t *ce;
> + vtd_re_t *re = phys_to_virt(vtd_root_table());
> +
> + assert(IS_ALIGNED(iova, SZ_4K));
> + assert(IS_ALIGNED(pa, SZ_4K));
> + assert(IS_ALIGNED(size, SZ_4K));
> +
> + bus_n = PCI_BDF_GET_BUS(sid);
> + devfn = PCI_BDF_GET_DEVFN(sid);
> +
> + /* Point to the correct root entry */
> + re += bus_n;
> +
> + if (!re->present) {
> + ce = alloc_page();
> + memset(ce, 0, PAGE_SIZE);
> + memset(re, 0, sizeof(*re));
> + re->context_table_p = virt_to_phys(ce) >> PAGE_SHIFT;
> + re->present = 1;
> + printf("allocated vt-d root entry for PCI bus %d\n",
> + bus_n);
> + } else
> + ce = VTD_FETCH_VIRT_ADDR(re->context_table_p);
> +
> + /* Point to the correct context entry */
> + ce += devfn;
> +
> + if (!ce->present) {
> + slptptr = alloc_page();
> + memset(slptptr, 0, PAGE_SIZE);
> + memset(ce, 0, sizeof(*ce));
> + /* To make it simple, domain ID is the same as SID */
> + ce->domain_id = sid;
> + /* We only test 39 bits width case (3-level paging) */
> + ce->addr_width = VTD_CE_AW_39BIT;
> + ce->slptptr = virt_to_phys(slptptr) >> PAGE_SHIFT;
> + ce->trans_type = VTD_CONTEXT_TT_MULTI_LEVEL;
> + ce->present = 1;
> + /* No error reporting yet */
> + ce->disable_fault_report = 1;
> + printf("allocated vt-d context entry for devfn 0x%x\n",
> + devfn);
> + } else
> + slptptr = VTD_FETCH_VIRT_ADDR(ce->slptptr);
> +
> + while (size) {
> + /* TODO: currently we only map 4K pages (level = 1) */
> + printf("map 4K page IOVA 0x%lx to 0x%lx (sid=0x%04x)\n",
> + iova, pa, sid);
> + vtd_install_pte(slptptr, iova, pa, 1);
> + size -= PAGE_SIZE;
> + iova += PAGE_SIZE;
> + pa += PAGE_SIZE;
> + }
> +}
> +
> void vtd_init(void)
> {
> setup_vm();
> diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
> index d95d76c..806d9fa 100644
> --- a/lib/x86/intel-iommu.h
> +++ b/lib/x86/intel-iommu.h
> @@ -20,6 +20,7 @@
> #include "isr.h"
> #include "smp.h"
> #include "desc.h"
> +#include "pci.h"
> #include "asm/io.h"
>
> #define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
> @@ -91,6 +92,27 @@
> #define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
> VTD_GCMD_SFL | VTD_GCMD_ROOT)
>
> +/* Supported Adjusted Guest Address Widths */
> +#define VTD_CAP_SAGAW_SHIFT 8
> + /* 39-bit AGAW, 3-level page-table */
> +#define VTD_CAP_SAGAW_39bit (0x2ULL << VTD_CAP_SAGAW_SHIFT)
> + /* 48-bit AGAW, 4-level page-table */
The leading space in the above comments is a bit strange
> +#define VTD_CAP_SAGAW_48bit (0x4ULL << VTD_CAP_SAGAW_SHIFT)
> +#define VTD_CAP_SAGAW VTD_CAP_SAGAW_39bit
> +
> +/* Both 1G/2M huge pages */
> +#define VTD_CAP_SLLPS ((1ULL << 34) | (1ULL << 35))
> +
> +#define VTD_CONTEXT_TT_MULTI_LEVEL 0
> +#define VTD_CONTEXT_TT_DEV_IOTLB 1
> +#define VTD_CONTEXT_TT_PASS_THROUGH 2
> +
> +#define VTD_PTE_R (1 << 0)
> +#define VTD_PTE_W (1 << 1)
> +#define VTD_PTE_RW (VTD_PTE_R | VTD_PTE_W)
> +#define VTD_PTE_ADDR GENMASK_ULL(51, 12)
> +#define VTD_PTE_HUGE (1 << 7)
> +
> #define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
>
> static inline void vtd_writel(unsigned int reg, uint32_t value)
> @@ -114,5 +136,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
> }
>
> void vtd_init(void);
> +void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
>
> #endif
> diff --git a/x86/Makefile.common b/x86/Makefile.common
> index 356d879..1dad18b 100644
> --- a/x86/Makefile.common
> +++ b/x86/Makefile.common
> @@ -3,6 +3,7 @@
> all: test_cases
>
> cflatobjs += lib/pci.o
> +cflatobjs += lib/pci-edu.o
> cflatobjs += lib/x86/io.o
> cflatobjs += lib/x86/smp.o
> cflatobjs += lib/x86/vm.o
> diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
> index f247913..755ccb1 100644
> --- a/x86/intel-iommu.c
> +++ b/x86/intel-iommu.c
> @@ -11,9 +11,49 @@
> */
>
> #include "intel-iommu.h"
> +#include "pci-edu.h"
> +
> +#define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
> +
> +void vtd_test_dmar(struct pci_edu_dev *dev)
> +{
> + void *page = alloc_page();
> +
> +#define DMA_TEST_WORD (0x12345678)
> + /* Modify the first 4 bytes of the page */
> + *(uint32_t *)page = DMA_TEST_WORD;
> +
> + /*
> + * Map the newly allocated page into IOVA address 0 (size 4K)
> + * of the device address space. Root entry and context entry
> + * will be automatically created when needed.
> + */
> + vtd_map_range(dev->pci_dev.bdf, 0, virt_to_phys(page), PAGE_SIZE);
> +
> + /*
> + * DMA the first 4 bytes of the page to EDU device buffer
> + * offset 0.
> + */
> + edu_dma(dev, 0, 4, 0, false);
missing blank line here
> + /*
> + * DMA the first 4 bytes of EDU device buffer into the page
> + * with offset 4 (so it'll be using 4-7 bytes).
> + */
> + edu_dma(dev, 4, 4, 0, true);
> +
> + /*
> + * Check data match between 0-3 bytes and 4-7 bytes of the
> + * page.
> + */
> + report(VTD_TEST_DMAR_4B, *((uint32_t *)page + 1) == DMA_TEST_WORD);
> +
> + free_page(page);
> +}
>
> int main(int argc, char *argv[])
> {
> + struct pci_edu_dev dev;
> +
> vtd_init();
>
> report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
> @@ -22,6 +62,16 @@ int main(int argc, char *argv[])
> report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
> report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
> report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
> + report("DMAR support 39 bits address width",
> + vtd_readq(DMAR_CAP_REG) & VTD_CAP_SAGAW);
> + report("DMAR support huge pages", vtd_readq(DMAR_CAP_REG) & VTD_CAP_SLLPS);
> +
> + if (!edu_init(&dev)) {
> + printf("Please specify \"-device edu\" to do "
> + "further IOMMU tests.\n");
> + report_skip(VTD_TEST_DMAR_4B);
> + } else
> + vtd_test_dmar(&dev);
>
> return report_summary();
> }
> --
> 2.7.4
>
Looks good to me from a framework/style perspective.
Thanks,
drew
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 15/17] pci: add msi support for 32/64bit address
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 15/17] pci: add msi support for 32/64bit address Peter Xu
@ 2016-11-10 20:10 ` Andrew Jones
2016-11-14 20:58 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 20:10 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:22AM -0500, Peter Xu wrote:
> pci_cap_walk() is provided to allow walk through all the capability bits
> for a specific PCI device. If a cap handler is provided, it'll be
> triggered if the cap is detected along the cap walk.
>
> MSI cap handler is the first one supported. We can add more cap handler
> in the future.
>
> Meanwhile, pci_setup_msi() API is provided to support basic 32/64 bit
> address MSI setup.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/pci.h | 3 +++
> 2 files changed, 67 insertions(+)
>
> diff --git a/lib/pci.c b/lib/pci.c
> index 971f02e..5b474f2 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -7,6 +7,69 @@
> #include "pci.h"
> #include "asm/pci.h"
>
> +typedef void (*pci_cap_handler)(struct pci_dev *dev, int cap_offset);
> +
> +static void pci_cap_msi_handler(struct pci_dev *dev, int cap_offset)
> +{
> + printf("Detected MSI for device 0x%x offset 0x%x\n",
> + dev->bdf, cap_offset);
> + dev->msi_offset = cap_offset;
> +}
> +
> +static pci_cap_handler cap_handlers[PCI_CAP_ID_MAX + 1] = {
> + [PCI_CAP_ID_MSI] = pci_cap_msi_handler,
> +};
> +
> +void pci_cap_walk(struct pci_dev *dev)
> +{
> + uint8_t cap_offset;
> + uint8_t cap_id;
> +
> + cap_offset = pci_config_readb(dev->bdf, PCI_CAPABILITY_LIST);
> + while (cap_offset) {
> + cap_id = pci_config_readb(dev->bdf, cap_offset);
> + printf("PCI detected cap 0x%x\n", cap_id);
> + if (cap_handlers[cap_id])
> + cap_handlers[cap_id](dev, cap_offset);
> + cap_offset = pci_config_readb(dev->bdf, cap_offset + 1);
> + }
> +}
> +
> +bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data)
> +{
> + uint16_t msi_control;
> + uint16_t offset;
> + pcidevaddr_t addr = dev->bdf;
> +
> + assert(dev);
If you're worried that dev might be null then you can't dereference
it the line above.
> +
> + if (!dev->msi_offset) {
> + printf("MSI: dev 0x%x does not support MSI.\n", addr);
> + return false;
> + }
> +
> + offset = dev->msi_offset;
> + msi_control = pci_config_readw(addr, offset + PCI_MSI_FLAGS);
> + pci_config_writel(addr, offset + PCI_MSI_ADDRESS_LO,
> + msi_addr & 0xffffffff);
> +
> + if (msi_control & PCI_MSI_FLAGS_64BIT) {
> + pci_config_writel(addr, offset + PCI_MSI_ADDRESS_HI,
> + (uint32_t)(msi_addr >> 32));
> + pci_config_writel(addr, offset + PCI_MSI_DATA_64, msi_data);
> + printf("MSI: dev 0x%x init 64bit address: ", addr);
> + } else {
> + pci_config_writel(addr, offset + PCI_MSI_DATA_32, msi_data);
> + printf("MSI: dev 0x%x init 32bit address: ", addr);
> + }
> + printf("addr=0x%lx, data=0x%x\n", msi_addr, msi_data);
> +
> + msi_control |= PCI_MSI_FLAGS_ENABLE;
> + pci_config_writew(addr, offset + PCI_MSI_FLAGS, msi_control);
> +
> + return true;
> +}
> +
> void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
> {
> uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
> @@ -255,5 +318,6 @@ int pci_enable_defaults(struct pci_dev *dev)
> pci_scan_bars(dev);
> /* Enable device DMA operations */
> pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
> + pci_cap_walk(dev);
> return 0;
> }
> diff --git a/lib/pci.h b/lib/pci.h
> index c4fef98..a5a1454 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -23,13 +23,16 @@ enum {
>
> struct pci_dev {
> uint16_t bdf;
> + uint16_t msi_offset;
> phys_addr_t bar[PCI_BAR_NUM];
> };
>
> void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> void pci_scan_bars(struct pci_dev *dev);
> void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> +void pci_cap_walk(struct pci_dev *dev);
> int pci_enable_defaults(struct pci_dev *dev);
> +bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data);
>
> typedef phys_addr_t iova_t;
>
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 16/17] x86: intel-iommu: add IR MSI test
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 16/17] x86: intel-iommu: add IR MSI test Peter Xu
@ 2016-11-10 20:18 ` Andrew Jones
0 siblings, 0 replies; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 20:18 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:23AM -0500, Peter Xu wrote:
> First of all, vtd_setup_msi() is provided. It setup IRTE entries,
> meanwhile, setup PCI device MSI vectors corresponding to VT-d spec.
>
> Meanwhile, IR MSI test is added to intel IOMMU unit test. The basic IR
> test is carried out by a edu INTR raise request. When write to the intr
> raise register, interrupt will be generated. Type of interrupt will
> depend on the setup (either INTx or MSI).
>
> Suggested-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci-edu.h | 1 +
> lib/x86/intel-iommu.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/x86/intel-iommu.h | 1 +
> x86/intel-iommu.c | 44 +++++++++++++++++++++++-
> 4 files changed, 139 insertions(+), 1 deletion(-)
>
> diff --git a/lib/pci-edu.h b/lib/pci-edu.h
> index af457df..5be278c 100644
> --- a/lib/pci-edu.h
> +++ b/lib/pci-edu.h
> @@ -32,6 +32,7 @@
> #define EDU_REG_ALIVE (0x4)
> #define EDU_REG_FACTORIAL (0x8)
> #define EDU_REG_STATUS (0x20)
> +#define EDU_REG_INTR_RAISE (0x60)
> #define EDU_REG_DMA_SRC (0x80)
> #define EDU_REG_DMA_DST (0x88)
> #define EDU_REG_DMA_COUNT (0x90)
> diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
> index 912a379..1d5f4b2 100644
> --- a/lib/x86/intel-iommu.c
> +++ b/lib/x86/intel-iommu.c
> @@ -12,6 +12,7 @@
>
> #include "intel-iommu.h"
> #include "libcflat.h"
> +#include "pci.h"
>
> /*
> * VT-d in QEMU currently only support 39 bits address width, which is
> @@ -48,6 +49,26 @@ struct vtd_context_entry {
> } __attribute__ ((packed));
> typedef struct vtd_context_entry vtd_ce_t;
>
> +struct vtd_irte {
> + uint32_t present:1;
> + uint32_t fault_disable:1; /* Fault Processing Disable */
> + uint32_t dest_mode:1; /* Destination Mode */
> + uint32_t redir_hint:1; /* Redirection Hint */
> + uint32_t trigger_mode:1; /* Trigger Mode */
> + uint32_t delivery_mode:3; /* Delivery Mode */
> + uint32_t __avail:4; /* Available spaces for software */
> + uint32_t __reserved_0:3; /* Reserved 0 */
> + uint32_t irte_mode:1; /* IRTE Mode */
> + uint32_t vector:8; /* Interrupt Vector */
> + uint32_t __reserved_1:8; /* Reserved 1 */
> + uint32_t dest_id; /* Destination ID */
> + uint16_t source_id:16; /* Source-ID */
> + uint64_t sid_q:2; /* Source-ID Qualifier */
> + uint64_t sid_vtype:2; /* Source-ID Validation Type */
> + uint64_t __reserved_2:44; /* Reserved 2 */
> +} __attribute__ ((packed));
> +typedef struct vtd_irte vtd_irte_t;
> +
> #define VTD_RTA_MASK (PAGE_MASK)
> #define VTD_IRTA_MASK (PAGE_MASK)
>
> @@ -205,6 +226,79 @@ void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
> }
> }
>
> +static uint16_t vtd_intr_index_alloc(void)
> +{
> + static int index_ctr = 0;
> + assert(index_ctr < 65535);
> + return index_ctr++;
> +}
> +
> +static void vtd_setup_irte(struct pci_dev *dev, vtd_irte_t *irte,
> + int vector, int dest_id)
> +{
> + assert(sizeof(vtd_irte_t) == 16);
> + memset(irte, 0, sizeof(*irte));
> + irte->fault_disable = 1;
> + irte->dest_mode = 0; /* physical */
> + irte->trigger_mode = 0; /* edge */
> + irte->delivery_mode = 0; /* fixed */
> + irte->irte_mode = 0; /* remapped */
> + irte->vector = vector;
> + irte->dest_id = dest_id;
> + irte->source_id = dev->bdf;
> + irte->sid_q = 0;
> + irte->sid_vtype = 1; /* full-sid verify */
> + irte->present = 1;
> +}
> +
> +struct vtd_msi_addr {
> + uint32_t __dont_care:2;
> + uint32_t handle_15:1; /* handle[15] */
> + uint32_t shv:1;
> + uint32_t interrupt_format:1;
> + uint32_t handle_0_14:15; /* handle[0:14] */
> + uint32_t head:12; /* 0xfee */
> + uint32_t addr_hi; /* not used except with x2apic */
> +} __attribute__ ((packed));
> +typedef struct vtd_msi_addr vtd_msi_addr_t;
> +
> +struct vtd_msi_data {
> + uint16_t __reserved;
> + uint16_t subhandle;
> +} __attribute__ ((packed));
> +typedef struct vtd_msi_data vtd_msi_data_t;
> +
> +/**
> + * vtd_setup_msi - setup MSI message for a device
> + *
> + * @dev: PCI device to setup MSI
> + * @vector: interrupt vector
> + * @dest_id: destination processor
> + */
> +bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id)
> +{
> + vtd_msi_data_t msi_data = {};
> + vtd_msi_addr_t msi_addr = {};
> + vtd_irte_t *irte = phys_to_virt(vtd_ir_table());
> + uint16_t index = vtd_intr_index_alloc();
> +
> + assert(sizeof(vtd_msi_addr_t) == 8);
> + assert(sizeof(vtd_msi_data_t) == 4);
Ideally, we'd have a BUILD_BUG type thing for this, but we don't,
so OK.
> +
> + printf("INTR: setup IRTE index %d\n", index);
> + vtd_setup_irte(dev, irte + index, vector, dest_id);
> +
> + msi_addr.handle_15 = index >> 15 & 1;
> + msi_addr.shv = 0;
> + msi_addr.interrupt_format = 1;
> + msi_addr.handle_0_14 = index & 0x7fff;
> + msi_addr.head = 0xfee;
> + msi_data.subhandle = 0;
> +
> + return pci_setup_msi(dev, *(uint64_t *)&msi_addr,
> + *(uint32_t *)&msi_data);
> +}
> +
> void vtd_init(void)
> {
> setup_vm();
> diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
> index 806d9fa..0725f59 100644
> --- a/lib/x86/intel-iommu.h
> +++ b/lib/x86/intel-iommu.h
> @@ -137,5 +137,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
>
> void vtd_init(void);
> void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
> +bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id);
>
> #endif
> diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
> index 755ccb1..cb9a991 100644
> --- a/x86/intel-iommu.c
> +++ b/x86/intel-iommu.c
> @@ -12,8 +12,10 @@
>
> #include "intel-iommu.h"
> #include "pci-edu.h"
> +#include "x86/apic.h"
>
> #define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
> +#define VTD_TEST_IR_MSI ("IR MSI")
>
> void vtd_test_dmar(struct pci_edu_dev *dev)
> {
> @@ -50,6 +52,43 @@ void vtd_test_dmar(struct pci_edu_dev *dev)
> free_page(page);
> }
>
> +static volatile bool edu_intr_recved;
> +
> +static void edu_isr(isr_regs_t *regs)
> +{
> + edu_intr_recved = true;
> + eoi();
> +}
> +
> +static void vtd_test_ir(struct pci_edu_dev *dev)
> +{
> +#define VTD_TEST_VECTOR (0xee)
> + /*
> + * Setup EDU PCI device MSI, using interrupt remapping. By
> + * default, EDU device is using INTx.
> + */
> + if (!vtd_setup_msi(&dev->pci_dev, VTD_TEST_VECTOR, 0)) {
> + printf("edu device does not support MSI, skip test\n");
> + report_skip(VTD_TEST_IR_MSI);
> + return;
> + }
> +
> + handle_irq(VTD_TEST_VECTOR, edu_isr);
> + irq_enable();
> +
> + /* Manually trigger INTR */
> + edu_reg_writel(dev, EDU_REG_INTR_RAISE, 1);
> +
> + while (!edu_intr_recved)
> + cpu_relax();
> +
> + /* Clear INTR bits */
> + edu_reg_writel(dev, EDU_REG_INTR_RAISE, 0);
> +
> + /* We are good as long as we reach here */
> + report(VTD_TEST_IR_MSI, true);
> +}
> +
> int main(int argc, char *argv[])
> {
> struct pci_edu_dev dev;
> @@ -70,8 +109,11 @@ int main(int argc, char *argv[])
> printf("Please specify \"-device edu\" to do "
> "further IOMMU tests.\n");
> report_skip(VTD_TEST_DMAR_4B);
> - } else
> + report_skip(VTD_TEST_IR_MSI);
> + } else {
> vtd_test_dmar(&dev);
> + vtd_test_ir(&dev);
> + }
>
> return report_summary();
> }
> --
> 2.7.4
>
I agree with the style/approach, but didn't open a spec.
drew
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 17/17] x86/unittests: add intel-iommu test
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 17/17] x86/unittests: add intel-iommu test Peter Xu
@ 2016-11-10 20:21 ` Andrew Jones
2016-11-14 21:07 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 20:21 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:24AM -0500, Peter Xu wrote:
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> x86/unittests.cfg | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/x86/unittests.cfg b/x86/unittests.cfg
> index 23395c6..5413838 100644
> --- a/x86/unittests.cfg
> +++ b/x86/unittests.cfg
> @@ -217,3 +217,10 @@ extra_params = -cpu kvm64,hv_time,hv_synic,hv_stimer -device hyperv-testdev
> file = hyperv_clock.flat
> smp = 2
> extra_params = -cpu kvm64,hv_time
> +
> +[intel_iommu]
> +file = intel-iommu.flat
> +arch = x86_64
> +timeout = 30
> +smp = 4
> +extra_params = -M q35,kernel-irqchip=split -device intel-iommu,intremap=on,eim=off -device edu
> --
> 2.7.4
>
I haven't looked up eim=off means, whether or not we want it, but
otherwise it looks good to me.
drew
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 00/17] VT-d unit test
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
` (17 preceding siblings ...)
2016-11-09 15:19 ` [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
@ 2016-11-10 20:25 ` Andrew Jones
2016-11-14 21:19 ` Peter Xu
18 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-10 20:25 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Wed, Nov 09, 2016 at 10:10:07AM -0500, Peter Xu wrote:
[...]
> Peter Xu (17):
> x86/asm: add cpu_relax()
> libcflat: introduce is_power_of_2()
> x86: intel-iommu: add vt-d init test
> libcflat: add IS_ALIGNED() macro, and page sizes
> libcflat: moving MIN/MAX here
> vm/page: provide PGDIR_OFFSET() macro
> pci: introduce struct pci_dev
> pci: provide pci_scan_bars()
> x86/vmexit: leverage pci_scan_bars()
> pci: provide pci_cmd_set_clr()
> pci: provide pci_enable_defaults()
> pci: add bdf helpers
> pci: edu: introduce pci-edu helpers
> x86: intel-iommu: add dmar test
> pci: add msi support for 32/64bit address
> x86: intel-iommu: add IR MSI test
> x86/unittests: add intel-iommu test
I think you can squash some of these patches together
03+17
08+09
10+11
12+14
Thanks,
drew
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 02/17] libcflat: introduce is_power_of_2()
2016-11-10 18:20 ` Andrew Jones
@ 2016-11-14 20:14 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:14 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 07:20:00PM +0100, Andrew Jones wrote:
> On Wed, Nov 09, 2016 at 10:10:09AM -0500, Peter Xu wrote:
> > Suggested-by: Andrew Jones <drjones@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> > lib/libcflat.h | 5 +++++
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/lib/libcflat.h b/lib/libcflat.h
> > index 72b1bf9..19bd0c6 100644
> > --- a/lib/libcflat.h
> > +++ b/lib/libcflat.h
> > @@ -103,4 +103,9 @@ do { \
> > } \
> > } while (0)
> >
> > +static inline bool is_power_of_2(unsigned long n)
> > +{
> > + return (n && !(n & (n - 1)));
>
> nit: outer () are unnecessary
Will fix.
>
> > +}
> > +
> > #endif
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 03/17] x86: intel-iommu: add vt-d init test
2016-11-10 19:09 ` Andrew Jones
@ 2016-11-14 20:18 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:18 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 08:09:36PM +0100, Andrew Jones wrote:
[...]
> > +static uint64_t vtd_root_table(void)
> > +{
> > + /* No extend root table support yet */
> > + return vtd_readq(DMAR_RTADDR_REG) & VTD_RTA_MASK;
>
> Should use tabs. You can run the kernel's scripts/checkpatch.pl,
> since we're using the same coding standard (pretty much...)
Yes, the indentation is wrong, which is to be fixed. Will try the
script later and see what else I can get.
>
> > +}
> > +
> > +static uint64_t vtd_ir_table(void)
> > +{
> > + return vtd_readq(DMAR_IRTA_REG) & VTD_IRTA_MASK;
> > +}
>
> The above two functions are both only used once each. Do we need them?
They are both used in further patches of this series. Also they might
be used in the future as well. So I'll keep them if you would not
mind.
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 07/17] pci: introduce struct pci_dev
2016-11-10 19:21 ` Andrew Jones
@ 2016-11-14 20:22 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:22 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 08:21:54PM +0100, Andrew Jones wrote:
[...]
> > /* Scan bus look for a specific device. Only bus 0 scanned for now. */
> > pcidevaddr_t pci_find_dev(uint16_t vendor_id, uint16_t device_id)
> > {
> > pcidevaddr_t dev;
> >
> > - for (dev = 0; dev < 256; ++dev) {
> > + for (dev = 0; dev < PCI_DEVFN_MAX; ++dev)
> > if (pci_config_readw(dev, PCI_VENDOR_ID) == vendor_id &&
> > pci_config_readw(dev, PCI_DEVICE_ID) == device_id)
> > return dev;
> > - }
>
> I liked the {} here because the "one" line spans three.
Sure. I'll keep them.
[...]
> > -void pci_bar_print(pcidevaddr_t dev, int bar_num)
> > +void pci_bar_print(struct pci_dev *dev, int bar_num)
> > {
> > phys_addr_t size, start, end;
> > uint32_t bar;
> > @@ -187,6 +195,9 @@ static void pci_dev_print(pcidevaddr_t dev)
> > uint8_t subclass = pci_config_readb(dev, PCI_CLASS_DEVICE);
> > uint8_t class = pci_config_readb(dev, PCI_CLASS_DEVICE + 1);
> > int i;
> > + struct pci_dev pci_dev;
>
> Putting that above the 'int i' would appease my aesthetics OCD...
Sure.
[...]
> Besides my nits
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars()
2016-11-10 19:24 ` Andrew Jones
@ 2016-11-14 20:33 ` Peter Xu
2016-11-14 21:18 ` Andrew Jones
0 siblings, 1 reply; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:33 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 08:24:19PM +0100, Andrew Jones wrote:
[...]
> > +void pci_scan_bars(struct pci_dev *dev)
> > +{
> > + int i = 0;
> > +
> > + for (i = 0; i < PCI_BAR_NUM; i++) {
> > + if (!pci_bar_is_valid(dev, i))
> > + continue;
> > + dev->bar[i] = pci_bar_get_addr(dev, i);
>
> What happens when you get_addr a 64-bit bar in the middle?
> Shouldn't we skip that?
Hmm yes... Do you like this?
---------8<----------
diff --git a/lib/pci.c b/lib/pci.c
index 0593699..2a58b30 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -230,11 +230,15 @@ void pci_print(void)
void pci_scan_bars(struct pci_dev *dev)
{
- int i = 0;
+ int i;
+
+ memset(&dev->bar[0], 0, sizeof(dev->bar));
for (i = 0; i < PCI_BAR_NUM; i++) {
if (!pci_bar_is_valid(dev, i))
continue;
dev->bar[i] = pci_bar_get_addr(dev, i);
+ if (pci_bar_is64(dev, i))
+ i++;
}
}
--------->8----------
pci_bar_is64() is called twice per bar, but I think it's okay here
since it's during init and the whole scan is called only once for 6
bars.
Thanks,
-- peterx
^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 09/17] x86/vmexit: leverage pci_scan_bars()
2016-11-10 19:27 ` Andrew Jones
@ 2016-11-14 20:35 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:35 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 08:27:58PM +0100, Andrew Jones wrote:
[...]
> > @@ -389,17 +389,12 @@ int main(int ac, char **av)
> > ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
> > if (ret != PCIDEVADDR_INVALID) {
> > pci_dev_init(&pcidev, ret);
> > - for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
> > - if (!pci_bar_is_valid(&pcidev, i)) {
> > - continue;
> > - }
> > - if (pci_bar_is_memory(&pcidev, i)) {
> > - membar = pci_bar_get_addr(&pcidev, i);
> > - pci_test.memaddr = ioremap(membar, PAGE_SIZE);
> > - } else {
> > - pci_test.iobar = pci_bar_get_addr(&pcidev, i);
> > - }
> > - }
> > + pci_scan_bars(&pcidev);
> > + assert(pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_MEM));
> > + assert(!pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_IO));
> > + membar = pcidev.bar[PCI_TESTDEV_BAR_MEM];
>
> nit: I'd drop 'membar' and just pass pcidev.bar to ioremap
Below printf() is using it as well. I kept it in case anyone is using
the below printf for any reason.
>
> > + pci_test.memaddr = ioremap(membar, PAGE_SIZE);
> > + pci_test.iobar = pcidev.bar[PCI_TESTDEV_BAR_IO];
> > printf("pci-testdev at 0x%x membar %lx iobar %x\n",
> > pcidev.bdf, membar, pci_test.iobar);
> > }
> > --
> > 2.7.4
> >
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 11/17] pci: provide pci_enable_defaults()
2016-11-10 19:33 ` Andrew Jones
@ 2016-11-14 20:42 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:42 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 08:33:42PM +0100, Andrew Jones wrote:
> On Wed, Nov 09, 2016 at 10:10:18AM -0500, Peter Xu wrote:
> > Provide a function to do most of the common PCI init work.
> >
> > Suggested-by: Andrew Jones <drjones@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> > lib/pci.c | 8 ++++++++
> > lib/pci.h | 1 +
> > 2 files changed, 9 insertions(+)
> >
> > diff --git a/lib/pci.c b/lib/pci.c
> > index fd17ea5..971f02e 100644
> > --- a/lib/pci.c
> > +++ b/lib/pci.c
> > @@ -249,3 +249,11 @@ void pci_scan_bars(struct pci_dev *dev)
> > dev->bar[i] = pci_bar_get_addr(dev, i);
> > }
> > }
> > +
> > +int pci_enable_defaults(struct pci_dev *dev)
> > +{
> > + pci_scan_bars(dev);
> > + /* Enable device DMA operations */
> > + pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
> > + return 0;
>
> Shouldn't this be a void function that just asserts on
> any errors it detects? I'm not sure why we're [currently
> unconditionally] returning zero.
This is my intention since I think this function might grow in the
future with lots of stuffs inside, and this function will need a
return code as long as any one of the future small functions will
return an error. However that's really not a strong reason to have
it... will remove it for your r-b. :-)
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 13/17] pci: edu: introduce pci-edu helpers
2016-11-10 19:45 ` Andrew Jones
@ 2016-11-14 20:48 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:48 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 08:45:03PM +0100, Andrew Jones wrote:
[...]
> General comment for whole series; I'm not a big fan of the
> () around the numeric constants. They're not necessary and
> starting to hurt my eyes :-)
Sorry for the unmeant hurt. :-(
I think I can remove them all in the whole series where apply (I hope
I won't miss any).
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 14/17] x86: intel-iommu: add dmar test
2016-11-10 19:53 ` Andrew Jones
@ 2016-11-14 20:54 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:54 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 08:53:02PM +0100, Andrew Jones wrote:
[...]
> > +/* Supported Adjusted Guest Address Widths */
> > +#define VTD_CAP_SAGAW_SHIFT 8
> > + /* 39-bit AGAW, 3-level page-table */
> > +#define VTD_CAP_SAGAW_39bit (0x2ULL << VTD_CAP_SAGAW_SHIFT)
> > + /* 48-bit AGAW, 4-level page-table */
>
> The leading space in the above comments is a bit strange
Yes, they are. :)
[...]
> > + /*
> > + * DMA the first 4 bytes of the page to EDU device buffer
> > + * offset 0.
> > + */
> > + edu_dma(dev, 0, 4, 0, false);
>
> missing blank line here
Will fix both. Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 15/17] pci: add msi support for 32/64bit address
2016-11-10 20:10 ` Andrew Jones
@ 2016-11-14 20:58 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 20:58 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 09:10:14PM +0100, Andrew Jones wrote:
[...]
> > +bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data)
> > +{
> > + uint16_t msi_control;
> > + uint16_t offset;
> > + pcidevaddr_t addr = dev->bdf;
> > +
> > + assert(dev);
>
> If you're worried that dev might be null then you can't dereference
> it the line above.
Right. Fixing up.
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 17/17] x86/unittests: add intel-iommu test
2016-11-10 20:21 ` Andrew Jones
@ 2016-11-14 21:07 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 21:07 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 09:21:41PM +0100, Andrew Jones wrote:
> On Wed, Nov 09, 2016 at 10:10:24AM -0500, Peter Xu wrote:
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> > x86/unittests.cfg | 7 +++++++
> > 1 file changed, 7 insertions(+)
> >
> > diff --git a/x86/unittests.cfg b/x86/unittests.cfg
> > index 23395c6..5413838 100644
> > --- a/x86/unittests.cfg
> > +++ b/x86/unittests.cfg
> > @@ -217,3 +217,10 @@ extra_params = -cpu kvm64,hv_time,hv_synic,hv_stimer -device hyperv-testdev
> > file = hyperv_clock.flat
> > smp = 2
> > extra_params = -cpu kvm64,hv_time
> > +
> > +[intel_iommu]
> > +file = intel-iommu.flat
> > +arch = x86_64
> > +timeout = 30
> > +smp = 4
> > +extra_params = -M q35,kernel-irqchip=split -device intel-iommu,intremap=on,eim=off -device edu
> > --
> > 2.7.4
> >
>
> I haven't looked up eim=off means, whether or not we want it, but
> otherwise it looks good to me.
When set eim=off, x2apic will be disabled. So basically "eim" bit is
to configure whether we want to support x2apic in the guest, in case
we want to use more than 255 vcpus.
Here:
- if with eim=off, vt-d test will need latest QEMU to run (so if with
older QEMUs, it'll fail)
- if without eim=off, vt-d test will need latest KVM to run (so if
with older KVM, it'll fail)
I just chose the 1st one since I guess a latest QEMU is easier to get
and compile than kernel.
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars()
2016-11-14 20:33 ` Peter Xu
@ 2016-11-14 21:18 ` Andrew Jones
2016-11-14 21:27 ` Peter Xu
0 siblings, 1 reply; 45+ messages in thread
From: Andrew Jones @ 2016-11-14 21:18 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Mon, Nov 14, 2016 at 03:33:22PM -0500, Peter Xu wrote:
> On Thu, Nov 10, 2016 at 08:24:19PM +0100, Andrew Jones wrote:
>
> [...]
>
> > > +void pci_scan_bars(struct pci_dev *dev)
> > > +{
> > > + int i = 0;
> > > +
> > > + for (i = 0; i < PCI_BAR_NUM; i++) {
> > > + if (!pci_bar_is_valid(dev, i))
> > > + continue;
> > > + dev->bar[i] = pci_bar_get_addr(dev, i);
> >
> > What happens when you get_addr a 64-bit bar in the middle?
> > Shouldn't we skip that?
>
> Hmm yes... Do you like this?
>
> ---------8<----------
>
> diff --git a/lib/pci.c b/lib/pci.c
> index 0593699..2a58b30 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -230,11 +230,15 @@ void pci_print(void)
>
> void pci_scan_bars(struct pci_dev *dev)
> {
> - int i = 0;
> + int i;
> +
> + memset(&dev->bar[0], 0, sizeof(dev->bar));
I'm not sure what this memset is for. Anyway, if you need to zero a
bar, why not just
bar = (phys_addr_t)0;
>
> for (i = 0; i < PCI_BAR_NUM; i++) {
> if (!pci_bar_is_valid(dev, i))
> continue;
> dev->bar[i] = pci_bar_get_addr(dev, i);
> + if (pci_bar_is64(dev, i))
> + i++;
> }
yes, this is consistent with how Alex did it in pci_probe. So
looks good to me.
> }
>
> --------->8----------
>
> pci_bar_is64() is called twice per bar, but I think it's okay here
> since it's during init and the whole scan is called only once for 6
> bars.
yeah, the "inefficiency" is no biggy.
Thanks,
drew
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 00/17] VT-d unit test
2016-11-10 20:25 ` Andrew Jones
@ 2016-11-14 21:19 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 21:19 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Thu, Nov 10, 2016 at 09:25:47PM +0100, Andrew Jones wrote:
> On Wed, Nov 09, 2016 at 10:10:07AM -0500, Peter Xu wrote:
> [...]
> > Peter Xu (17):
> > x86/asm: add cpu_relax()
> > libcflat: introduce is_power_of_2()
> > x86: intel-iommu: add vt-d init test
> > libcflat: add IS_ALIGNED() macro, and page sizes
> > libcflat: moving MIN/MAX here
> > vm/page: provide PGDIR_OFFSET() macro
> > pci: introduce struct pci_dev
> > pci: provide pci_scan_bars()
> > x86/vmexit: leverage pci_scan_bars()
> > pci: provide pci_cmd_set_clr()
> > pci: provide pci_enable_defaults()
> > pci: add bdf helpers
> > pci: edu: introduce pci-edu helpers
> > x86: intel-iommu: add dmar test
> > pci: add msi support for 32/64bit address
> > x86: intel-iommu: add IR MSI test
> > x86/unittests: add intel-iommu test
>
> I think you can squash some of these patches together
> 03+17
> 08+09
> 10+11
> 12+14
These all looks sane. Will apply in the next post.
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars()
2016-11-14 21:18 ` Andrew Jones
@ 2016-11-14 21:27 ` Peter Xu
0 siblings, 0 replies; 45+ messages in thread
From: Peter Xu @ 2016-11-14 21:27 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, rkrcmar, agordeev, jan.kiszka, pbonzini
On Mon, Nov 14, 2016 at 10:18:34PM +0100, Andrew Jones wrote:
[...]
> > diff --git a/lib/pci.c b/lib/pci.c
> > index 0593699..2a58b30 100644
> > --- a/lib/pci.c
> > +++ b/lib/pci.c
> > @@ -230,11 +230,15 @@ void pci_print(void)
> >
> > void pci_scan_bars(struct pci_dev *dev)
> > {
> > - int i = 0;
> > + int i;
> > +
> > + memset(&dev->bar[0], 0, sizeof(dev->bar));
>
> I'm not sure what this memset is for. Anyway, if you need to zero a
> bar, why not just
>
> bar = (phys_addr_t)0;
The memset is to zero all bars before hand. But I think bar =
(phys_addr_t)0 works enough here:
void pci_scan_bars(struct pci_dev *dev)
{
int i;
for (i = 0; i < PCI_BAR_NUM; i++) {
if (!pci_bar_is_valid(dev, i))
continue;
dev->bar[i] = pci_bar_get_addr(dev, i);
if (pci_bar_is64(dev, i)) {
i++;
dev->bar[i] = (phys_addr_t)0;
}
}
}
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 45+ messages in thread
end of thread, other threads:[~2016-11-14 21:27 UTC | newest]
Thread overview: 45+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-09 15:10 [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 01/17] x86/asm: add cpu_relax() Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 02/17] libcflat: introduce is_power_of_2() Peter Xu
2016-11-10 18:20 ` Andrew Jones
2016-11-14 20:14 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 03/17] x86: intel-iommu: add vt-d init test Peter Xu
2016-11-10 19:09 ` Andrew Jones
2016-11-14 20:18 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 04/17] libcflat: add IS_ALIGNED() macro, and page sizes Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 05/17] libcflat: moving MIN/MAX here Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 06/17] vm/page: provide PGDIR_OFFSET() macro Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 07/17] pci: introduce struct pci_dev Peter Xu
2016-11-10 19:21 ` Andrew Jones
2016-11-14 20:22 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 08/17] pci: provide pci_scan_bars() Peter Xu
2016-11-10 19:24 ` Andrew Jones
2016-11-14 20:33 ` Peter Xu
2016-11-14 21:18 ` Andrew Jones
2016-11-14 21:27 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 09/17] x86/vmexit: leverage pci_scan_bars() Peter Xu
2016-11-10 19:27 ` Andrew Jones
2016-11-14 20:35 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 10/17] pci: provide pci_cmd_set_clr() Peter Xu
2016-11-10 19:31 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 11/17] pci: provide pci_enable_defaults() Peter Xu
2016-11-10 19:33 ` Andrew Jones
2016-11-14 20:42 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 12/17] pci: add bdf helpers Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 13/17] pci: edu: introduce pci-edu helpers Peter Xu
2016-11-10 19:45 ` Andrew Jones
2016-11-14 20:48 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 14/17] x86: intel-iommu: add dmar test Peter Xu
2016-11-10 19:53 ` Andrew Jones
2016-11-14 20:54 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 15/17] pci: add msi support for 32/64bit address Peter Xu
2016-11-10 20:10 ` Andrew Jones
2016-11-14 20:58 ` Peter Xu
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 16/17] x86: intel-iommu: add IR MSI test Peter Xu
2016-11-10 20:18 ` Andrew Jones
2016-11-09 15:10 ` [PATCH kvm-unit-tests v2 17/17] x86/unittests: add intel-iommu test Peter Xu
2016-11-10 20:21 ` Andrew Jones
2016-11-14 21:07 ` Peter Xu
2016-11-09 15:19 ` [PATCH kvm-unit-tests v2 00/17] VT-d unit test Peter Xu
2016-11-10 20:25 ` Andrew Jones
2016-11-14 21:19 ` Peter Xu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.