* [PATCH v2 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM
@ 2020-05-08 17:30 Eric Auger
2020-05-08 17:30 ` [PATCH v2 1/5] qdev: Introduce DEFINE_PROP_RESERVED_REGION Eric Auger
` (4 more replies)
0 siblings, 5 replies; 15+ messages in thread
From: Eric Auger @ 2020-05-08 17:30 UTC (permalink / raw)
To: eric.auger.pro, eric.auger, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, bbhushan2, peterx, armbru, pbonzini
By default the virtio-iommu translates MSI transactions. This
behavior is inherited from ARM SMMU. However the virt machine
code knows where the MSI doorbells are, so we can easily
declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that
setting the guest iommu subsystem will not need to map MSIs.
This setup will simplify the VFIO integration.
In this series, the ITS or GICV2M doorbells are declared as
HW MSI regions to be bypassed by the VIRTIO-IOMMU.
This also paves the way to the x86 integration where the MSI
region, [0xFEE00000,0xFEEFFFFF], will be exposed by the q35
machine. However this will be handled in a separate series
when not-DT support gets resolved.
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v5.0.0-virtio-iommu-msi-bypass-v2
History:
v1 -> v2:
- check which MSI controller is in use and advertise the
corresponding MSI doorbell
- managed for both ITS and GICv2M
- various fixes spotted by Peter and Jean-Philippe, see
individual logs
v1: Most of those patches were respinned from
[PATCH for-5.0 v11 00/20] VIRTIO-IOMMU device
except the last one which is new
Eric Auger (5):
qdev: Introduce DEFINE_PROP_RESERVED_REGION
virtio-iommu: Implement RESV_MEM probe request
virtio-iommu: Handle reserved regions in the translation process
virtio-iommu-pci: Add array of Interval properties
hw/arm/virt: Let the virtio-iommu bypass MSIs
include/exec/memory.h | 6 ++
include/hw/arm/virt.h | 6 ++
include/hw/qdev-properties.h | 3 +
include/hw/virtio/virtio-iommu.h | 2 +
include/qemu/typedefs.h | 1 +
hw/arm/virt.c | 18 +++++
hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++
hw/virtio/virtio-iommu-pci.c | 3 +
hw/virtio/virtio-iommu.c | 114 +++++++++++++++++++++++++++++--
hw/virtio/trace-events | 1 +
10 files changed, 239 insertions(+), 4 deletions(-)
--
2.20.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v2 1/5] qdev: Introduce DEFINE_PROP_RESERVED_REGION
2020-05-08 17:30 [PATCH v2 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM Eric Auger
@ 2020-05-08 17:30 ` Eric Auger
2020-05-08 17:30 ` [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request Eric Auger
` (3 subsequent siblings)
4 siblings, 0 replies; 15+ messages in thread
From: Eric Auger @ 2020-05-08 17:30 UTC (permalink / raw)
To: eric.auger.pro, eric.auger, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, bbhushan2, peterx, armbru, pbonzini
Introduce a new property defining a reserved region:
<low address>, <high address>, <type>.
This will be used to encode reserved IOVA regions.
For instance, in virtio-iommu use case, reserved IOVA regions
will be passed by the machine code to the virtio-iommu-pci
device (an array of those). The type of the reserved region
will match the virtio_iommu_probe_resv_mem subtype value:
- VIRTIO_IOMMU_RESV_MEM_T_RESERVED (0)
- VIRTIO_IOMMU_RESV_MEM_T_MSI (1)
on PC/Q35 machine, this will be used to inform the
virtio-iommu-pci device it should bypass the MSI region.
The reserved region will be: 0xfee00000, 0xfeefffff, 1.
On ARM, we can declare the ITS MSI doorbell as an MSI
region to prevent MSIs from being mapped on guest side.
Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
v11 -> v12:
- rename into DEFINE_PROP_RESERVED_REGION
- do not use g_strsplit anymore, use endptr instead
- remove 0x references
---
include/exec/memory.h | 6 +++
include/hw/qdev-properties.h | 3 ++
include/qemu/typedefs.h | 1 +
hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++++++++++++++
4 files changed, 99 insertions(+)
diff --git a/include/exec/memory.h b/include/exec/memory.h
index e000bd2f97..7e47afabe8 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -57,6 +57,12 @@ struct MemoryRegionMmio {
CPUWriteMemoryFunc *write[3];
};
+struct ReservedRegion {
+ hwaddr low;
+ hwaddr high;
+ unsigned int type;
+};
+
typedef struct IOMMUTLBEntry IOMMUTLBEntry;
/* See address_space_translate: bit 0 is read, bit 1 is write. */
diff --git a/include/hw/qdev-properties.h b/include/hw/qdev-properties.h
index f161604fb6..03bf850a7e 100644
--- a/include/hw/qdev-properties.h
+++ b/include/hw/qdev-properties.h
@@ -19,6 +19,7 @@ extern const PropertyInfo qdev_prop_string;
extern const PropertyInfo qdev_prop_chr;
extern const PropertyInfo qdev_prop_tpm;
extern const PropertyInfo qdev_prop_macaddr;
+extern const PropertyInfo qdev_prop_reserved_region;
extern const PropertyInfo qdev_prop_on_off_auto;
extern const PropertyInfo qdev_prop_multifd_compression;
extern const PropertyInfo qdev_prop_losttickpolicy;
@@ -183,6 +184,8 @@ extern const PropertyInfo qdev_prop_pcie_link_width;
DEFINE_PROP(_n, _s, _f, qdev_prop_drive_iothread, BlockBackend *)
#define DEFINE_PROP_MACADDR(_n, _s, _f) \
DEFINE_PROP(_n, _s, _f, qdev_prop_macaddr, MACAddr)
+#define DEFINE_PROP_RESERVED_REGION(_n, _s, _f) \
+ DEFINE_PROP(_n, _s, _f, qdev_prop_reserved_region, ReservedRegion)
#define DEFINE_PROP_ON_OFF_AUTO(_n, _s, _f, _d) \
DEFINE_PROP_SIGNED(_n, _s, _f, _d, qdev_prop_on_off_auto, OnOffAuto)
#define DEFINE_PROP_MULTIFD_COMPRESSION(_n, _s, _f, _d) \
diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
index ecf3cde26c..85c4f891f4 100644
--- a/include/qemu/typedefs.h
+++ b/include/qemu/typedefs.h
@@ -59,6 +59,7 @@ typedef struct ISABus ISABus;
typedef struct ISADevice ISADevice;
typedef struct IsaDma IsaDma;
typedef struct MACAddr MACAddr;
+typedef struct ReservedRegion ReservedRegion;
typedef struct MachineClass MachineClass;
typedef struct MachineState MachineState;
typedef struct MemoryListener MemoryListener;
diff --git a/hw/core/qdev-properties.c b/hw/core/qdev-properties.c
index 2047114fca..c2e0cc7cda 100644
--- a/hw/core/qdev-properties.c
+++ b/hw/core/qdev-properties.c
@@ -14,6 +14,7 @@
#include "qapi/visitor.h"
#include "chardev/char.h"
#include "qemu/uuid.h"
+#include "qemu/cutils.h"
void qdev_prop_set_after_realize(DeviceState *dev, const char *name,
Error **errp)
@@ -577,6 +578,94 @@ const PropertyInfo qdev_prop_macaddr = {
.set = set_mac,
};
+/* --- Reserved Region --- */
+
+/*
+ * accepted syntax version:
+ * <low address>,<high address>,<type>
+ * where low/high addresses are uint64_t in hexadecimal
+ * and type is an unsigned integer in decimal
+ */
+static void get_reserved_region(Object *obj, Visitor *v, const char *name,
+ void *opaque, Error **errp)
+{
+ DeviceState *dev = DEVICE(obj);
+ Property *prop = opaque;
+ ReservedRegion *rr = qdev_get_prop_ptr(dev, prop);
+ char buffer[64];
+ char *p = buffer;
+
+ snprintf(buffer, sizeof(buffer), "0x%"PRIx64",0x%"PRIx64",%u",
+ rr->low, rr->high, rr->type);
+
+ visit_type_str(v, name, &p, errp);
+}
+
+static void set_reserved_region(Object *obj, Visitor *v, const char *name,
+ void *opaque, Error **errp)
+{
+ DeviceState *dev = DEVICE(obj);
+ Property *prop = opaque;
+ ReservedRegion *rr = qdev_get_prop_ptr(dev, prop);
+ Error *local_err = NULL;
+ const char *endptr;
+ char *str;
+ int ret;
+
+ if (dev->realized) {
+ qdev_prop_set_after_realize(dev, name, errp);
+ return;
+ }
+
+ visit_type_str(v, name, &str, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ return;
+ }
+
+ ret = qemu_strtou64(str, &endptr, 16, &rr->low);
+ if (ret) {
+ error_setg(errp, "Failed to decode reserved region low addr");
+ error_append_hint(errp,
+ "should be an address in hexadecimal\n");
+ goto out;
+ }
+ if (*endptr != ',') {
+ goto separator_error;
+ }
+
+ ret = qemu_strtou64(endptr + 1, &endptr, 16, &rr->high);
+ if (ret) {
+ error_setg(errp, "Failed to decode reserved region high addr");
+ error_append_hint(errp,
+ "should be an address in hexadecimal\n");
+ goto out;
+ }
+ if (*endptr != ',') {
+ goto separator_error;
+ }
+
+ ret = qemu_strtoui(endptr + 1, &endptr, 10, &rr->type);
+ if (ret) {
+ error_setg(errp, "Failed to decode reserved region type");
+ error_append_hint(errp, "should be an unsigned integer in decimal\n");
+ }
+ goto out;
+
+separator_error:
+ error_setg(errp, "reserved region fields must be separated with commas");
+out:
+ g_free(str);
+ return;
+}
+
+const PropertyInfo qdev_prop_reserved_region = {
+ .name = "reserved_region",
+ .description = "Reserved Region, example: 0xFEE00000,0xFEEFFFFF,0",
+ .get = get_reserved_region,
+ .set = set_reserved_region,
+};
+
/* --- on/off/auto --- */
const PropertyInfo qdev_prop_on_off_auto = {
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-08 17:30 [PATCH v2 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM Eric Auger
2020-05-08 17:30 ` [PATCH v2 1/5] qdev: Introduce DEFINE_PROP_RESERVED_REGION Eric Auger
@ 2020-05-08 17:30 ` Eric Auger
2020-05-11 6:38 ` [EXT] " Bharat Bhushan
2020-05-08 17:30 ` [PATCH v2 3/5] virtio-iommu: Handle reserved regions in the translation process Eric Auger
` (2 subsequent siblings)
4 siblings, 1 reply; 15+ messages in thread
From: Eric Auger @ 2020-05-08 17:30 UTC (permalink / raw)
To: eric.auger.pro, eric.auger, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, bbhushan2, peterx, armbru, pbonzini
This patch implements the PROBE request. At the moment,
only THE RESV_MEM property is handled. The first goal is
to report iommu wide reserved regions such as the MSI regions
set by the machine code. On x86 this will be the IOAPIC MSI
region, [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS
doorbell.
In the future we may introduce per device reserved regions.
This will be useful when protecting host assigned devices
which may expose their own reserved regions
Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
v1 -> v2:
- move the unlock back to the same place
- remove the push label and factorize the code after the out label
- fix a bunch of cpu_to_leX according to the latest spec revision
- do not remove sizeof(last) from free space
- check the ep exists
---
include/hw/virtio/virtio-iommu.h | 2 +
hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
hw/virtio/trace-events | 1 +
3 files changed, 93 insertions(+), 4 deletions(-)
diff --git a/include/hw/virtio/virtio-iommu.h b/include/hw/virtio/virtio-iommu.h
index e653004d7c..49eb105cd8 100644
--- a/include/hw/virtio/virtio-iommu.h
+++ b/include/hw/virtio/virtio-iommu.h
@@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
GHashTable *as_by_busptr;
IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
PCIBus *primary_bus;
+ ReservedRegion *reserved_regions;
+ uint32_t nb_reserved_regions;
GTree *domains;
QemuMutex mutex;
GTree *endpoints;
diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c
index 22ba8848c2..35d772e021 100644
--- a/hw/virtio/virtio-iommu.c
+++ b/hw/virtio/virtio-iommu.c
@@ -38,6 +38,7 @@
/* Max size */
#define VIOMMU_DEFAULT_QUEUE_SIZE 256
+#define VIOMMU_PROBE_SIZE 512
typedef struct VirtIOIOMMUDomain {
uint32_t id;
@@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
return ret;
}
+static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t ep,
+ uint8_t *buf, size_t free)
+{
+ struct virtio_iommu_probe_resv_mem prop = {};
+ size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
+ int i;
+
+ total = size * s->nb_reserved_regions;
+
+ if (total > free) {
+ return -ENOSPC;
+ }
+
+ for (i = 0; i < s->nb_reserved_regions; i++) {
+ prop.head.type = cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
+ prop.head.length = cpu_to_le16(length);
+ prop.subtype = s->reserved_regions[i].type;
+ prop.start = cpu_to_le64(s->reserved_regions[i].low);
+ prop.end = cpu_to_le64(s->reserved_regions[i].high);
+
+ memcpy(buf, &prop, size);
+
+ trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
+ prop.start, prop.end);
+ buf += size;
+ }
+ return total;
+}
+
+/**
+ * virtio_iommu_probe - Fill the probe request buffer with
+ * the properties the device is able to return and add a NONE
+ * property at the end.
+ */
+static int virtio_iommu_probe(VirtIOIOMMU *s,
+ struct virtio_iommu_req_probe *req,
+ uint8_t *buf)
+{
+ uint32_t ep_id = le32_to_cpu(req->endpoint);
+ size_t free = VIOMMU_PROBE_SIZE;
+ ssize_t count;
+
+ if (!virtio_iommu_mr(s, ep_id)) {
+ return VIRTIO_IOMMU_S_NOENT;
+ }
+
+ count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
+ if (count < 0) {
+ return VIRTIO_IOMMU_S_INVAL;
+ }
+ buf += count;
+ free -= count;
+
+ /* Fill the rest with zeroes */
+ memset(buf, 0, free);
+
+ return VIRTIO_IOMMU_S_OK;
+}
+
static int virtio_iommu_iov_to_req(struct iovec *iov,
unsigned int iov_cnt,
void *req, size_t req_sz)
@@ -407,15 +467,27 @@ virtio_iommu_handle_req(detach)
virtio_iommu_handle_req(map)
virtio_iommu_handle_req(unmap)
+static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
+ struct iovec *iov,
+ unsigned int iov_cnt,
+ uint8_t *buf)
+{
+ struct virtio_iommu_req_probe req;
+ int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req, sizeof(req));
+
+ return ret ? ret : virtio_iommu_probe(s, &req, buf);
+}
+
static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
{
VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
struct virtio_iommu_req_head head;
struct virtio_iommu_req_tail tail = {};
+ size_t output_size = sizeof(tail), sz;
VirtQueueElement *elem;
unsigned int iov_cnt;
struct iovec *iov;
- size_t sz;
+ void *buf = NULL;
for (;;) {
elem = virtqueue_pop(vq, sizeof(VirtQueueElement));
@@ -452,6 +524,17 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
case VIRTIO_IOMMU_T_UNMAP:
tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
break;
+ case VIRTIO_IOMMU_T_PROBE:
+ {
+ struct virtio_iommu_req_tail *ptail;
+
+ output_size = s->config.probe_size + sizeof(tail);
+ buf = g_malloc0(output_size);
+
+ ptail = (struct virtio_iommu_req_tail *)
+ (buf + s->config.probe_size);
+ ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
+ }
default:
tail.status = VIRTIO_IOMMU_S_UNSUPP;
}
@@ -459,12 +542,13 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
out:
sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
- &tail, sizeof(tail));
- assert(sz == sizeof(tail));
+ buf ? buf : &tail, output_size);
+ assert(sz == output_size);
- virtqueue_push(vq, elem, sizeof(tail));
+ virtqueue_push(vq, elem, sz);
virtio_notify(vdev, vq);
g_free(elem);
+ g_free(buf);
}
}
@@ -667,6 +751,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
s->config.page_size_mask = TARGET_PAGE_MASK;
s->config.input_range.end = -1UL;
s->config.domain_range.end = 32;
+ s->config.probe_size = VIOMMU_PROBE_SIZE;
virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC);
@@ -676,6 +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
+ virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
qemu_mutex_init(&s->mutex);
diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
index e83500bee9..5550475691 100644
--- a/hw/virtio/trace-events
+++ b/hw/virtio/trace-events
@@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc domain=%d"
virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64
+virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype, uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64" end=0x%"PRIx64
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 3/5] virtio-iommu: Handle reserved regions in the translation process
2020-05-08 17:30 [PATCH v2 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM Eric Auger
2020-05-08 17:30 ` [PATCH v2 1/5] qdev: Introduce DEFINE_PROP_RESERVED_REGION Eric Auger
2020-05-08 17:30 ` [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request Eric Auger
@ 2020-05-08 17:30 ` Eric Auger
2020-05-11 21:11 ` Peter Xu
2020-05-08 17:30 ` [PATCH v2 4/5] virtio-iommu-pci: Add array of Interval properties Eric Auger
2020-05-08 17:30 ` [PATCH v2 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs Eric Auger
4 siblings, 1 reply; 15+ messages in thread
From: Eric Auger @ 2020-05-08 17:30 UTC (permalink / raw)
To: eric.auger.pro, eric.auger, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, bbhushan2, peterx, armbru, pbonzini
When translating an address we need to check if it belongs to
a reserved virtual address range. If it does, there are 2 cases:
- it belongs to a RESERVED region: the guest should neither use
this address in a MAP not instruct the end-point to DMA on
them. We report an error
- It belongs to an MSI region: we bypass the translation.
Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
v1 -> v2:
- use addr when testing addr belongs to the reserved region
and use a block local variable
---
hw/virtio/virtio-iommu.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c
index 35d772e021..ba72cfaa63 100644
--- a/hw/virtio/virtio-iommu.c
+++ b/hw/virtio/virtio-iommu.c
@@ -607,6 +607,7 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr,
uint32_t sid, flags;
bool bypass_allowed;
bool found;
+ int i;
interval.low = addr;
interval.high = addr + 1;
@@ -640,6 +641,25 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr,
goto unlock;
}
+ for (i = 0; i < s->nb_reserved_regions; i++) {
+ ReservedRegion *reg = &s->reserved_regions[i];
+
+ if (addr >= reg->low && addr <= reg->high) {
+ switch (reg->type) {
+ case VIRTIO_IOMMU_RESV_MEM_T_MSI:
+ entry.perm = flag;
+ break;
+ case VIRTIO_IOMMU_RESV_MEM_T_RESERVED:
+ default:
+ virtio_iommu_report_fault(s, VIRTIO_IOMMU_FAULT_R_MAPPING,
+ VIRTIO_IOMMU_FAULT_F_ADDRESS,
+ sid, addr);
+ break;
+ }
+ goto unlock;
+ }
+ }
+
if (!ep->domain) {
if (!bypass_allowed) {
error_report_once("%s %02x:%02x.%01x not attached to any domain",
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 4/5] virtio-iommu-pci: Add array of Interval properties
2020-05-08 17:30 [PATCH v2 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM Eric Auger
` (2 preceding siblings ...)
2020-05-08 17:30 ` [PATCH v2 3/5] virtio-iommu: Handle reserved regions in the translation process Eric Auger
@ 2020-05-08 17:30 ` Eric Auger
2020-05-08 17:30 ` [PATCH v2 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs Eric Auger
4 siblings, 0 replies; 15+ messages in thread
From: Eric Auger @ 2020-05-08 17:30 UTC (permalink / raw)
To: eric.auger.pro, eric.auger, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, bbhushan2, peterx, armbru, pbonzini
The machine may need to pass reserved regions to the
virtio-iommu-pci device (such as the MSI window on x86).
So let's add an array of Interval properties.
Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
v12 -> v12:
- added Jean's R-b
---
hw/virtio/virtio-iommu-pci.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/hw/virtio/virtio-iommu-pci.c b/hw/virtio/virtio-iommu-pci.c
index 3dfbf55b47..44ae9ebc11 100644
--- a/hw/virtio/virtio-iommu-pci.c
+++ b/hw/virtio/virtio-iommu-pci.c
@@ -33,6 +33,9 @@ struct VirtIOIOMMUPCI {
static Property virtio_iommu_pci_properties[] = {
DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
+ DEFINE_PROP_ARRAY("reserved-regions", VirtIOIOMMUPCI,
+ vdev.nb_reserved_regions, vdev.reserved_regions,
+ qdev_prop_reserved_region, ReservedRegion),
DEFINE_PROP_END_OF_LIST(),
};
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs
2020-05-08 17:30 [PATCH v2 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM Eric Auger
` (3 preceding siblings ...)
2020-05-08 17:30 ` [PATCH v2 4/5] virtio-iommu-pci: Add array of Interval properties Eric Auger
@ 2020-05-08 17:30 ` Eric Auger
2020-05-22 14:43 ` Jean-Philippe Brucker
4 siblings, 1 reply; 15+ messages in thread
From: Eric Auger @ 2020-05-08 17:30 UTC (permalink / raw)
To: eric.auger.pro, eric.auger, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, bbhushan2, peterx, armbru, pbonzini
At the moment the virtio-iommu translates MSI transactions.
This behavior is inherited from ARM SMMU. The virt machine
code knows where the guest MSI doorbells are so we can easily
declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that
setting the guest will not map MSIs through the IOMMU and those
transactions will be simply bypassed.
Depending on which MSI controller is in use (ITS or GICV2M),
we declare either:
- the ITS interrupt translation space (ITS_base + 0x10000),
containing the GITS_TRANSLATOR or
- The GICV2M single frame, containing the MSI_SETSP_NS register.
Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
v1 -> v2:
- Test which MSI controller is instantiated
- If GICV2M is in use, declare its doorbell as an MSI doorbell too
---
include/hw/arm/virt.h | 6 ++++++
hw/arm/virt.c | 18 ++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
index 6d67ace76e..ad20cb6e15 100644
--- a/include/hw/arm/virt.h
+++ b/include/hw/arm/virt.h
@@ -96,6 +96,11 @@ typedef enum VirtIOMMUType {
VIRT_IOMMU_VIRTIO,
} VirtIOMMUType;
+typedef enum VirtMSIControllerType {
+ VIRT_GICV2M,
+ VIRT_ITS,
+} VirtMSIControllerType;
+
typedef enum VirtGICType {
VIRT_GIC_VERSION_MAX,
VIRT_GIC_VERSION_HOST,
@@ -135,6 +140,7 @@ typedef struct {
OnOffAuto acpi;
VirtGICType gic_version;
VirtIOMMUType iommu;
+ VirtMSIControllerType msi_controller;
uint16_t virtio_iommu_bdf;
struct arm_boot_info bootinfo;
MemMapEntry *memmap;
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 634db0cfe9..d2dd07885b 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -602,6 +602,7 @@ static void create_its(VirtMachineState *vms)
sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, vms->memmap[VIRT_GIC_ITS].base);
fdt_add_its_gic_node(vms);
+ vms->msi_controller = VIRT_ITS;
}
static void create_v2m(VirtMachineState *vms)
@@ -622,6 +623,7 @@ static void create_v2m(VirtMachineState *vms)
}
fdt_add_v2m_gic_node(vms);
+ vms->msi_controller = VIRT_GICV2M;
}
static void create_gic(VirtMachineState *vms)
@@ -2136,8 +2138,24 @@ out:
static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
DeviceState *dev, Error **errp)
{
+ VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+
if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
virt_memory_pre_plug(hotplug_dev, dev, errp);
+ } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
+ /* we declare a VIRTIO_IOMMU_RESV_MEM_T_MSI region */
+
+ if (vms->msi_controller == VIRT_ITS) {
+ /* GITS_TRANSLATER page */
+ qdev_prop_set_uint32(dev, "len-reserved-regions", 1);
+ qdev_prop_set_string(dev, "reserved-regions[0]",
+ "0x8090000, 0x809FFFF, 1");
+ } else if (vms->msi_controller == VIRT_GICV2M) {
+ /* MSI_SETSPI_NS page */
+ qdev_prop_set_uint32(dev, "len-reserved-regions", 1);
+ qdev_prop_set_string(dev, "reserved-regions[0]",
+ "0x8020000, 0x8020FFF, 1");
+ }
}
}
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* RE: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-08 17:30 ` [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request Eric Auger
@ 2020-05-11 6:38 ` Bharat Bhushan
2020-05-11 6:55 ` Auger Eric
0 siblings, 1 reply; 15+ messages in thread
From: Bharat Bhushan @ 2020-05-11 6:38 UTC (permalink / raw)
To: Eric Auger, eric.auger.pro, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, peterx, armbru, pbonzini
Hi Eric,
> -----Original Message-----
> From: Eric Auger <eric.auger@redhat.com>
> Sent: Friday, May 8, 2020 11:01 PM
> To: eric.auger.pro@gmail.com; eric.auger@redhat.com; qemu-devel@nongnu.org;
> qemu-arm@nongnu.org; peter.maydell@linaro.org; mst@redhat.com; jean-
> philippe@linaro.org; Bharat Bhushan <bbhushan2@marvell.com>;
> peterx@redhat.com; armbru@redhat.com; pbonzini@redhat.com
> Subject: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
>
> External Email
>
> ----------------------------------------------------------------------
> This patch implements the PROBE request. At the moment, only THE RESV_MEM
> property is handled. The first goal is to report iommu wide reserved regions such as
> the MSI regions set by the machine code. On x86 this will be the IOAPIC MSI region,
> [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
>
> In the future we may introduce per device reserved regions.
> This will be useful when protecting host assigned devices which may expose their
> own reserved regions
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
>
> v1 -> v2:
> - move the unlock back to the same place
> - remove the push label and factorize the code after the out label
> - fix a bunch of cpu_to_leX according to the latest spec revision
> - do not remove sizeof(last) from free space
> - check the ep exists
> ---
> include/hw/virtio/virtio-iommu.h | 2 +
> hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
> hw/virtio/trace-events | 1 +
> 3 files changed, 93 insertions(+), 4 deletions(-)
>
> diff --git a/include/hw/virtio/virtio-iommu.h b/include/hw/virtio/virtio-iommu.h
> index e653004d7c..49eb105cd8 100644
> --- a/include/hw/virtio/virtio-iommu.h
> +++ b/include/hw/virtio/virtio-iommu.h
> @@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
> GHashTable *as_by_busptr;
> IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
> PCIBus *primary_bus;
> + ReservedRegion *reserved_regions;
> + uint32_t nb_reserved_regions;
> GTree *domains;
> QemuMutex mutex;
> GTree *endpoints;
> diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c index
> 22ba8848c2..35d772e021 100644
> --- a/hw/virtio/virtio-iommu.c
> +++ b/hw/virtio/virtio-iommu.c
> @@ -38,6 +38,7 @@
>
> /* Max size */
> #define VIOMMU_DEFAULT_QUEUE_SIZE 256
> +#define VIOMMU_PROBE_SIZE 512
>
> typedef struct VirtIOIOMMUDomain {
> uint32_t id;
> @@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
> return ret;
> }
>
> +static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t ep,
> + uint8_t *buf, size_t
> +free) {
> + struct virtio_iommu_probe_resv_mem prop = {};
> + size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
> + int i;
> +
> + total = size * s->nb_reserved_regions;
> +
> + if (total > free) {
> + return -ENOSPC;
> + }
> +
> + for (i = 0; i < s->nb_reserved_regions; i++) {
> + prop.head.type = cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
> + prop.head.length = cpu_to_le16(length);
> + prop.subtype = s->reserved_regions[i].type;
> + prop.start = cpu_to_le64(s->reserved_regions[i].low);
> + prop.end = cpu_to_le64(s->reserved_regions[i].high);
> +
> + memcpy(buf, &prop, size);
> +
> + trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
> + prop.start, prop.end);
> + buf += size;
> + }
> + return total;
> +}
> +
> +/**
> + * virtio_iommu_probe - Fill the probe request buffer with
> + * the properties the device is able to return and add a NONE
> + * property at the end.
> + */
> +static int virtio_iommu_probe(VirtIOIOMMU *s,
> + struct virtio_iommu_req_probe *req,
> + uint8_t *buf) {
> + uint32_t ep_id = le32_to_cpu(req->endpoint);
> + size_t free = VIOMMU_PROBE_SIZE;
> + ssize_t count;
> +
> + if (!virtio_iommu_mr(s, ep_id)) {
> + return VIRTIO_IOMMU_S_NOENT;
> + }
> +
> + count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
> + if (count < 0) {
> + return VIRTIO_IOMMU_S_INVAL;
> + }
> + buf += count;
> + free -= count;
> +
> + /* Fill the rest with zeroes */
> + memset(buf, 0, free);
No need to fill with zero here as "buf" is set to zero on allocation, no?
Thanks
-Bharat
> +
> + return VIRTIO_IOMMU_S_OK;
> +}
> +
> static int virtio_iommu_iov_to_req(struct iovec *iov,
> unsigned int iov_cnt,
> void *req, size_t req_sz) @@ -407,15 +467,27 @@
> virtio_iommu_handle_req(detach)
> virtio_iommu_handle_req(map)
> virtio_iommu_handle_req(unmap)
>
> +static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
> + struct iovec *iov,
> + unsigned int iov_cnt,
> + uint8_t *buf) {
> + struct virtio_iommu_req_probe req;
> + int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req, sizeof(req));
> +
> + return ret ? ret : virtio_iommu_probe(s, &req, buf); }
> +
> static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq) {
> VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
> struct virtio_iommu_req_head head;
> struct virtio_iommu_req_tail tail = {};
> + size_t output_size = sizeof(tail), sz;
> VirtQueueElement *elem;
> unsigned int iov_cnt;
> struct iovec *iov;
> - size_t sz;
> + void *buf = NULL;
>
> for (;;) {
> elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); @@ -452,6 +524,17 @@
> static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
> case VIRTIO_IOMMU_T_UNMAP:
> tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
> break;
> + case VIRTIO_IOMMU_T_PROBE:
> + {
> + struct virtio_iommu_req_tail *ptail;
> +
> + output_size = s->config.probe_size + sizeof(tail);
> + buf = g_malloc0(output_size);
> +
> + ptail = (struct virtio_iommu_req_tail *)
> + (buf + s->config.probe_size);
> + ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
> + }
> default:
> tail.status = VIRTIO_IOMMU_S_UNSUPP;
> }
> @@ -459,12 +542,13 @@ static void
> virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
>
> out:
> sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
> - &tail, sizeof(tail));
> - assert(sz == sizeof(tail));
> + buf ? buf : &tail, output_size);
> + assert(sz == output_size);
>
> - virtqueue_push(vq, elem, sizeof(tail));
> + virtqueue_push(vq, elem, sz);
> virtio_notify(vdev, vq);
> g_free(elem);
> + g_free(buf);
> }
> }
>
> @@ -667,6 +751,7 @@ static void virtio_iommu_device_realize(DeviceState *dev,
> Error **errp)
> s->config.page_size_mask = TARGET_PAGE_MASK;
> s->config.input_range.end = -1UL;
> s->config.domain_range.end = 32;
> + s->config.probe_size = VIOMMU_PROBE_SIZE;
>
> virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
> virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC); @@ -676,6
> +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error
> **errp)
> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
> + virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
>
> qemu_mutex_init(&s->mutex);
>
> diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index
> e83500bee9..5550475691 100644
> --- a/hw/virtio/trace-events
> +++ b/hw/virtio/trace-events
> @@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc
> domain=%d"
> virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
> virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid)
> "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
> virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint,
> uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64
> +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype,
> +uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64"
> +end=0x%"PRIx64
> --
> 2.20.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-11 6:38 ` [EXT] " Bharat Bhushan
@ 2020-05-11 6:55 ` Auger Eric
2020-05-11 8:42 ` Bharat Bhushan
0 siblings, 1 reply; 15+ messages in thread
From: Auger Eric @ 2020-05-11 6:55 UTC (permalink / raw)
To: Bharat Bhushan, eric.auger.pro, qemu-devel, qemu-arm,
peter.maydell, mst, jean-philippe, peterx, armbru, pbonzini
Hi Bharat,
On 5/11/20 8:38 AM, Bharat Bhushan wrote:
> Hi Eric,
>
>> -----Original Message-----
>> From: Eric Auger <eric.auger@redhat.com>
>> Sent: Friday, May 8, 2020 11:01 PM
>> To: eric.auger.pro@gmail.com; eric.auger@redhat.com; qemu-devel@nongnu.org;
>> qemu-arm@nongnu.org; peter.maydell@linaro.org; mst@redhat.com; jean-
>> philippe@linaro.org; Bharat Bhushan <bbhushan2@marvell.com>;
>> peterx@redhat.com; armbru@redhat.com; pbonzini@redhat.com
>> Subject: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
>>
>> External Email
>>
>> ----------------------------------------------------------------------
>> This patch implements the PROBE request. At the moment, only THE RESV_MEM
>> property is handled. The first goal is to report iommu wide reserved regions such as
>> the MSI regions set by the machine code. On x86 this will be the IOAPIC MSI region,
>> [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
>>
>> In the future we may introduce per device reserved regions.
>> This will be useful when protecting host assigned devices which may expose their
>> own reserved regions
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>>
>> v1 -> v2:
>> - move the unlock back to the same place
>> - remove the push label and factorize the code after the out label
>> - fix a bunch of cpu_to_leX according to the latest spec revision
>> - do not remove sizeof(last) from free space
>> - check the ep exists
>> ---
>> include/hw/virtio/virtio-iommu.h | 2 +
>> hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
>> hw/virtio/trace-events | 1 +
>> 3 files changed, 93 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/hw/virtio/virtio-iommu.h b/include/hw/virtio/virtio-iommu.h
>> index e653004d7c..49eb105cd8 100644
>> --- a/include/hw/virtio/virtio-iommu.h
>> +++ b/include/hw/virtio/virtio-iommu.h
>> @@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
>> GHashTable *as_by_busptr;
>> IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
>> PCIBus *primary_bus;
>> + ReservedRegion *reserved_regions;
>> + uint32_t nb_reserved_regions;
>> GTree *domains;
>> QemuMutex mutex;
>> GTree *endpoints;
>> diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c index
>> 22ba8848c2..35d772e021 100644
>> --- a/hw/virtio/virtio-iommu.c
>> +++ b/hw/virtio/virtio-iommu.c
>> @@ -38,6 +38,7 @@
>>
>> /* Max size */
>> #define VIOMMU_DEFAULT_QUEUE_SIZE 256
>> +#define VIOMMU_PROBE_SIZE 512
>>
>> typedef struct VirtIOIOMMUDomain {
>> uint32_t id;
>> @@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
>> return ret;
>> }
>>
>> +static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t ep,
>> + uint8_t *buf, size_t
>> +free) {
>> + struct virtio_iommu_probe_resv_mem prop = {};
>> + size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
>> + int i;
>> +
>> + total = size * s->nb_reserved_regions;
>> +
>> + if (total > free) {
>> + return -ENOSPC;
>> + }
>> +
>> + for (i = 0; i < s->nb_reserved_regions; i++) {
>> + prop.head.type = cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
>> + prop.head.length = cpu_to_le16(length);
>> + prop.subtype = s->reserved_regions[i].type;
>> + prop.start = cpu_to_le64(s->reserved_regions[i].low);
>> + prop.end = cpu_to_le64(s->reserved_regions[i].high);
>> +
>> + memcpy(buf, &prop, size);
>> +
>> + trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
>> + prop.start, prop.end);
>> + buf += size;
>> + }
>> + return total;
>> +}
>> +
>> +/**
>> + * virtio_iommu_probe - Fill the probe request buffer with
>> + * the properties the device is able to return and add a NONE
>> + * property at the end.
>> + */
>> +static int virtio_iommu_probe(VirtIOIOMMU *s,
>> + struct virtio_iommu_req_probe *req,
>> + uint8_t *buf) {
>> + uint32_t ep_id = le32_to_cpu(req->endpoint);
>> + size_t free = VIOMMU_PROBE_SIZE;
>> + ssize_t count;
>> +
>> + if (!virtio_iommu_mr(s, ep_id)) {
>> + return VIRTIO_IOMMU_S_NOENT;
>> + }
>> +
>> + count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
>> + if (count < 0) {
>> + return VIRTIO_IOMMU_S_INVAL;
>> + }
>> + buf += count;
>> + free -= count;
>> +
>> + /* Fill the rest with zeroes */
>> + memset(buf, 0, free);
>
> No need to fill with zero here as "buf" is set to zero on allocation, no?
You're right. I will remove this in the next version.
Thanks
Eric
>
> Thanks
> -Bharat
>
>> +
>> + return VIRTIO_IOMMU_S_OK;
>> +}
>> +
>> static int virtio_iommu_iov_to_req(struct iovec *iov,
>> unsigned int iov_cnt,
>> void *req, size_t req_sz) @@ -407,15 +467,27 @@
>> virtio_iommu_handle_req(detach)
>> virtio_iommu_handle_req(map)
>> virtio_iommu_handle_req(unmap)
>>
>> +static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
>> + struct iovec *iov,
>> + unsigned int iov_cnt,
>> + uint8_t *buf) {
>> + struct virtio_iommu_req_probe req;
>> + int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req, sizeof(req));
>> +
>> + return ret ? ret : virtio_iommu_probe(s, &req, buf); }
>> +
>> static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq) {
>> VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
>> struct virtio_iommu_req_head head;
>> struct virtio_iommu_req_tail tail = {};
>> + size_t output_size = sizeof(tail), sz;
>> VirtQueueElement *elem;
>> unsigned int iov_cnt;
>> struct iovec *iov;
>> - size_t sz;
>> + void *buf = NULL;
>>
>> for (;;) {
>> elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); @@ -452,6 +524,17 @@
>> static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
>> case VIRTIO_IOMMU_T_UNMAP:
>> tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
>> break;
>> + case VIRTIO_IOMMU_T_PROBE:
>> + {
>> + struct virtio_iommu_req_tail *ptail;
>> +
>> + output_size = s->config.probe_size + sizeof(tail);
>> + buf = g_malloc0(output_size);
>> +
>> + ptail = (struct virtio_iommu_req_tail *)
>> + (buf + s->config.probe_size);
>> + ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
>> + }
>> default:
>> tail.status = VIRTIO_IOMMU_S_UNSUPP;
>> }
>> @@ -459,12 +542,13 @@ static void
>> virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
>>
>> out:
>> sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
>> - &tail, sizeof(tail));
>> - assert(sz == sizeof(tail));
>> + buf ? buf : &tail, output_size);
>> + assert(sz == output_size);
>>
>> - virtqueue_push(vq, elem, sizeof(tail));
>> + virtqueue_push(vq, elem, sz);
>> virtio_notify(vdev, vq);
>> g_free(elem);
>> + g_free(buf);
>> }
>> }
>>
>> @@ -667,6 +751,7 @@ static void virtio_iommu_device_realize(DeviceState *dev,
>> Error **errp)
>> s->config.page_size_mask = TARGET_PAGE_MASK;
>> s->config.input_range.end = -1UL;
>> s->config.domain_range.end = 32;
>> + s->config.probe_size = VIOMMU_PROBE_SIZE;
>>
>> virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
>> virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC); @@ -676,6
>> +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error
>> **errp)
>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
>> + virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
>>
>> qemu_mutex_init(&s->mutex);
>>
>> diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index
>> e83500bee9..5550475691 100644
>> --- a/hw/virtio/trace-events
>> +++ b/hw/virtio/trace-events
>> @@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc
>> domain=%d"
>> virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
>> virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid)
>> "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
>> virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint,
>> uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64
>> +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype,
>> +uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64"
>> +end=0x%"PRIx64
>> --
>> 2.20.1
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-11 6:55 ` Auger Eric
@ 2020-05-11 8:42 ` Bharat Bhushan
2020-05-11 8:49 ` Auger Eric
0 siblings, 1 reply; 15+ messages in thread
From: Bharat Bhushan @ 2020-05-11 8:42 UTC (permalink / raw)
To: Auger Eric, eric.auger.pro, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, peterx, armbru, pbonzini
Hi Eric,
> -----Original Message-----
> From: Auger Eric <eric.auger@redhat.com>
> Sent: Monday, May 11, 2020 12:26 PM
> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
> armbru@redhat.com; pbonzini@redhat.com
> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe
> request
>
> Hi Bharat,
> On 5/11/20 8:38 AM, Bharat Bhushan wrote:
> > Hi Eric,
> >
> >> -----Original Message-----
> >> From: Eric Auger <eric.auger@redhat.com>
> >> Sent: Friday, May 8, 2020 11:01 PM
> >> To: eric.auger.pro@gmail.com; eric.auger@redhat.com;
> >> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
> >> mst@redhat.com; jean- philippe@linaro.org; Bharat Bhushan
> >> <bbhushan2@marvell.com>; peterx@redhat.com; armbru@redhat.com;
> >> pbonzini@redhat.com
> >> Subject: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe
> >> request
> >>
> >> External Email
> >>
> >> ---------------------------------------------------------------------
> >> - This patch implements the PROBE request. At the moment, only THE
> >> RESV_MEM property is handled. The first goal is to report iommu wide
> >> reserved regions such as the MSI regions set by the machine code. On
> >> x86 this will be the IOAPIC MSI region,
> >> [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
> >>
> >> In the future we may introduce per device reserved regions.
> >> This will be useful when protecting host assigned devices which may
> >> expose their own reserved regions
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >>
> >> v1 -> v2:
> >> - move the unlock back to the same place
> >> - remove the push label and factorize the code after the out label
> >> - fix a bunch of cpu_to_leX according to the latest spec revision
> >> - do not remove sizeof(last) from free space
> >> - check the ep exists
> >> ---
> >> include/hw/virtio/virtio-iommu.h | 2 +
> >> hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
> >> hw/virtio/trace-events | 1 +
> >> 3 files changed, 93 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/include/hw/virtio/virtio-iommu.h
> >> b/include/hw/virtio/virtio-iommu.h
> >> index e653004d7c..49eb105cd8 100644
> >> --- a/include/hw/virtio/virtio-iommu.h
> >> +++ b/include/hw/virtio/virtio-iommu.h
> >> @@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
> >> GHashTable *as_by_busptr;
> >> IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
> >> PCIBus *primary_bus;
> >> + ReservedRegion *reserved_regions;
> >> + uint32_t nb_reserved_regions;
> >> GTree *domains;
> >> QemuMutex mutex;
> >> GTree *endpoints;
> >> diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c
> >> index
> >> 22ba8848c2..35d772e021 100644
> >> --- a/hw/virtio/virtio-iommu.c
> >> +++ b/hw/virtio/virtio-iommu.c
> >> @@ -38,6 +38,7 @@
> >>
> >> /* Max size */
> >> #define VIOMMU_DEFAULT_QUEUE_SIZE 256
> >> +#define VIOMMU_PROBE_SIZE 512
> >>
> >> typedef struct VirtIOIOMMUDomain {
> >> uint32_t id;
> >> @@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
> >> return ret;
> >> }
> >>
> >> +static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t ep,
> >> + uint8_t *buf, size_t
> >> +free) {
> >> + struct virtio_iommu_probe_resv_mem prop = {};
> >> + size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
> >> + int i;
> >> +
> >> + total = size * s->nb_reserved_regions;
> >> +
> >> + if (total > free) {
> >> + return -ENOSPC;
> >> + }
> >> +
> >> + for (i = 0; i < s->nb_reserved_regions; i++) {
> >> + prop.head.type = cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
> >> + prop.head.length = cpu_to_le16(length);
> >> + prop.subtype = s->reserved_regions[i].type;
> >> + prop.start = cpu_to_le64(s->reserved_regions[i].low);
> >> + prop.end = cpu_to_le64(s->reserved_regions[i].high);
> >> +
> >> + memcpy(buf, &prop, size);
> >> +
> >> + trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
> >> + prop.start, prop.end);
> >> + buf += size;
> >> + }
> >> + return total;
> >> +}
> >> +
> >> +/**
> >> + * virtio_iommu_probe - Fill the probe request buffer with
> >> + * the properties the device is able to return and add a NONE
> >> + * property at the end.
> >> + */
> >> +static int virtio_iommu_probe(VirtIOIOMMU *s,
> >> + struct virtio_iommu_req_probe *req,
> >> + uint8_t *buf) {
> >> + uint32_t ep_id = le32_to_cpu(req->endpoint);
> >> + size_t free = VIOMMU_PROBE_SIZE;
> >> + ssize_t count;
> >> +
> >> + if (!virtio_iommu_mr(s, ep_id)) {
> >> + return VIRTIO_IOMMU_S_NOENT;
> >> + }
> >> +
> >> + count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
> >> + if (count < 0) {
> >> + return VIRTIO_IOMMU_S_INVAL;
> >> + }
> >> + buf += count;
> >> + free -= count;
> >> +
> >> + /* Fill the rest with zeroes */
> >> + memset(buf, 0, free);
> >
> > No need to fill with zero here as "buf" is set to zero on allocation, no?
>
> You're right. I will remove this in the next version.
>
> Thanks
>
> Eric
> >
> > Thanks
> > -Bharat
> >
> >> +
> >> + return VIRTIO_IOMMU_S_OK;
> >> +}
> >> +
> >> static int virtio_iommu_iov_to_req(struct iovec *iov,
> >> unsigned int iov_cnt,
> >> void *req, size_t req_sz) @@
> >> -407,15 +467,27 @@
> >> virtio_iommu_handle_req(detach)
> >> virtio_iommu_handle_req(map)
> >> virtio_iommu_handle_req(unmap)
> >>
> >> +static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
> >> + struct iovec *iov,
> >> + unsigned int iov_cnt,
> >> + uint8_t *buf) {
> >> + struct virtio_iommu_req_probe req;
> >> + int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req,
> >> +sizeof(req));
> >> +
> >> + return ret ? ret : virtio_iommu_probe(s, &req, buf); }
> >> +
> >> static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue
> *vq) {
> >> VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
> >> struct virtio_iommu_req_head head;
> >> struct virtio_iommu_req_tail tail = {};
> >> + size_t output_size = sizeof(tail), sz;
> >> VirtQueueElement *elem;
> >> unsigned int iov_cnt;
> >> struct iovec *iov;
> >> - size_t sz;
> >> + void *buf = NULL;
> >>
> >> for (;;) {
> >> elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); @@
> >> -452,6 +524,17 @@ static void virtio_iommu_handle_command(VirtIODevice
> *vdev, VirtQueue *vq)
> >> case VIRTIO_IOMMU_T_UNMAP:
> >> tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
> >> break;
> >> + case VIRTIO_IOMMU_T_PROBE:
As per spec
"
If the device does not offer the VIRTIO_IOMMU_F_PROBE feature, and if the driver sends a VIRTIO_-
IOMMU_T_PROBE request, then the device SHOULD NOT write the buffer and SHOULD set the used
length to zero.
"
So we should check if device supports "VIRTIO_IOMMU_F_PROBE" before proceed?
Thanks
-Bharat
> >> + {
> >> + struct virtio_iommu_req_tail *ptail;
> >> +
> >> + output_size = s->config.probe_size + sizeof(tail);
> >> + buf = g_malloc0(output_size);
> >> +
> >> + ptail = (struct virtio_iommu_req_tail *)
> >> + (buf + s->config.probe_size);
> >> + ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
> >> + }
> >> default:
> >> tail.status = VIRTIO_IOMMU_S_UNSUPP;
> >> }
> >> @@ -459,12 +542,13 @@ static void
> >> virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
> >>
> >> out:
> >> sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
> >> - &tail, sizeof(tail));
> >> - assert(sz == sizeof(tail));
> >> + buf ? buf : &tail, output_size);
> >> + assert(sz == output_size);
> >>
> >> - virtqueue_push(vq, elem, sizeof(tail));
> >> + virtqueue_push(vq, elem, sz);
> >> virtio_notify(vdev, vq);
> >> g_free(elem);
> >> + g_free(buf);
> >> }
> >> }
> >>
> >> @@ -667,6 +751,7 @@ static void
> >> virtio_iommu_device_realize(DeviceState *dev, Error **errp)
> >> s->config.page_size_mask = TARGET_PAGE_MASK;
> >> s->config.input_range.end = -1UL;
> >> s->config.domain_range.end = 32;
> >> + s->config.probe_size = VIOMMU_PROBE_SIZE;
> >>
> >> virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
> >> virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC);
> >> @@ -676,6
> >> +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev,
> >> +Error
> >> **errp)
> >> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
> >> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
> >> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
> >> + virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
> >>
> >> qemu_mutex_init(&s->mutex);
> >>
> >> diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index
> >> e83500bee9..5550475691 100644
> >> --- a/hw/virtio/trace-events
> >> +++ b/hw/virtio/trace-events
> >> @@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc
> >> domain=%d"
> >> virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
> >> virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr,
> >> uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
> >> virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t
> >> endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d
> >> address =0x%"PRIx64
> >> +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype,
> >> +uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64"
> >> +end=0x%"PRIx64
> >> --
> >> 2.20.1
> >
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-11 8:42 ` Bharat Bhushan
@ 2020-05-11 8:49 ` Auger Eric
2020-05-12 3:03 ` Bharat Bhushan
0 siblings, 1 reply; 15+ messages in thread
From: Auger Eric @ 2020-05-11 8:49 UTC (permalink / raw)
To: Bharat Bhushan, eric.auger.pro, qemu-devel, qemu-arm,
peter.maydell, mst, jean-philippe, peterx, armbru, pbonzini
Hi Bharat,
On 5/11/20 10:42 AM, Bharat Bhushan wrote:
> Hi Eric,
>
>> -----Original Message-----
>> From: Auger Eric <eric.auger@redhat.com>
>> Sent: Monday, May 11, 2020 12:26 PM
>> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
>> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
>> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
>> armbru@redhat.com; pbonzini@redhat.com
>> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe
>> request
>>
>> Hi Bharat,
>> On 5/11/20 8:38 AM, Bharat Bhushan wrote:
>>> Hi Eric,
>>>
>>>> -----Original Message-----
>>>> From: Eric Auger <eric.auger@redhat.com>
>>>> Sent: Friday, May 8, 2020 11:01 PM
>>>> To: eric.auger.pro@gmail.com; eric.auger@redhat.com;
>>>> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
>>>> mst@redhat.com; jean- philippe@linaro.org; Bharat Bhushan
>>>> <bbhushan2@marvell.com>; peterx@redhat.com; armbru@redhat.com;
>>>> pbonzini@redhat.com
>>>> Subject: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe
>>>> request
>>>>
>>>> External Email
>>>>
>>>> ---------------------------------------------------------------------
>>>> - This patch implements the PROBE request. At the moment, only THE
>>>> RESV_MEM property is handled. The first goal is to report iommu wide
>>>> reserved regions such as the MSI regions set by the machine code. On
>>>> x86 this will be the IOAPIC MSI region,
>>>> [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
>>>>
>>>> In the future we may introduce per device reserved regions.
>>>> This will be useful when protecting host assigned devices which may
>>>> expose their own reserved regions
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>>
>>>> v1 -> v2:
>>>> - move the unlock back to the same place
>>>> - remove the push label and factorize the code after the out label
>>>> - fix a bunch of cpu_to_leX according to the latest spec revision
>>>> - do not remove sizeof(last) from free space
>>>> - check the ep exists
>>>> ---
>>>> include/hw/virtio/virtio-iommu.h | 2 +
>>>> hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
>>>> hw/virtio/trace-events | 1 +
>>>> 3 files changed, 93 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/include/hw/virtio/virtio-iommu.h
>>>> b/include/hw/virtio/virtio-iommu.h
>>>> index e653004d7c..49eb105cd8 100644
>>>> --- a/include/hw/virtio/virtio-iommu.h
>>>> +++ b/include/hw/virtio/virtio-iommu.h
>>>> @@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
>>>> GHashTable *as_by_busptr;
>>>> IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
>>>> PCIBus *primary_bus;
>>>> + ReservedRegion *reserved_regions;
>>>> + uint32_t nb_reserved_regions;
>>>> GTree *domains;
>>>> QemuMutex mutex;
>>>> GTree *endpoints;
>>>> diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c
>>>> index
>>>> 22ba8848c2..35d772e021 100644
>>>> --- a/hw/virtio/virtio-iommu.c
>>>> +++ b/hw/virtio/virtio-iommu.c
>>>> @@ -38,6 +38,7 @@
>>>>
>>>> /* Max size */
>>>> #define VIOMMU_DEFAULT_QUEUE_SIZE 256
>>>> +#define VIOMMU_PROBE_SIZE 512
>>>>
>>>> typedef struct VirtIOIOMMUDomain {
>>>> uint32_t id;
>>>> @@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
>>>> return ret;
>>>> }
>>>>
>>>> +static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t ep,
>>>> + uint8_t *buf, size_t
>>>> +free) {
>>>> + struct virtio_iommu_probe_resv_mem prop = {};
>>>> + size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
>>>> + int i;
>>>> +
>>>> + total = size * s->nb_reserved_regions;
>>>> +
>>>> + if (total > free) {
>>>> + return -ENOSPC;
>>>> + }
>>>> +
>>>> + for (i = 0; i < s->nb_reserved_regions; i++) {
>>>> + prop.head.type = cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
>>>> + prop.head.length = cpu_to_le16(length);
>>>> + prop.subtype = s->reserved_regions[i].type;
>>>> + prop.start = cpu_to_le64(s->reserved_regions[i].low);
>>>> + prop.end = cpu_to_le64(s->reserved_regions[i].high);
>>>> +
>>>> + memcpy(buf, &prop, size);
>>>> +
>>>> + trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
>>>> + prop.start, prop.end);
>>>> + buf += size;
>>>> + }
>>>> + return total;
>>>> +}
>>>> +
>>>> +/**
>>>> + * virtio_iommu_probe - Fill the probe request buffer with
>>>> + * the properties the device is able to return and add a NONE
>>>> + * property at the end.
>>>> + */
>>>> +static int virtio_iommu_probe(VirtIOIOMMU *s,
>>>> + struct virtio_iommu_req_probe *req,
>>>> + uint8_t *buf) {
>>>> + uint32_t ep_id = le32_to_cpu(req->endpoint);
>>>> + size_t free = VIOMMU_PROBE_SIZE;
>>>> + ssize_t count;
>>>> +
>>>> + if (!virtio_iommu_mr(s, ep_id)) {
>>>> + return VIRTIO_IOMMU_S_NOENT;
>>>> + }
>>>> +
>>>> + count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
>>>> + if (count < 0) {
>>>> + return VIRTIO_IOMMU_S_INVAL;
>>>> + }
>>>> + buf += count;
>>>> + free -= count;
>>>> +
>>>> + /* Fill the rest with zeroes */
>>>> + memset(buf, 0, free);
>>>
>>> No need to fill with zero here as "buf" is set to zero on allocation, no?
>>
>> You're right. I will remove this in the next version.
>>
>> Thanks
>>
>> Eric
>>>
>>> Thanks
>>> -Bharat
>>>
>>>> +
>>>> + return VIRTIO_IOMMU_S_OK;
>>>> +}
>>>> +
>>>> static int virtio_iommu_iov_to_req(struct iovec *iov,
>>>> unsigned int iov_cnt,
>>>> void *req, size_t req_sz) @@
>>>> -407,15 +467,27 @@
>>>> virtio_iommu_handle_req(detach)
>>>> virtio_iommu_handle_req(map)
>>>> virtio_iommu_handle_req(unmap)
>>>>
>>>> +static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
>>>> + struct iovec *iov,
>>>> + unsigned int iov_cnt,
>>>> + uint8_t *buf) {
>>>> + struct virtio_iommu_req_probe req;
>>>> + int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req,
>>>> +sizeof(req));
>>>> +
>>>> + return ret ? ret : virtio_iommu_probe(s, &req, buf); }
>>>> +
>>>> static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue
>> *vq) {
>>>> VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
>>>> struct virtio_iommu_req_head head;
>>>> struct virtio_iommu_req_tail tail = {};
>>>> + size_t output_size = sizeof(tail), sz;
>>>> VirtQueueElement *elem;
>>>> unsigned int iov_cnt;
>>>> struct iovec *iov;
>>>> - size_t sz;
>>>> + void *buf = NULL;
>>>>
>>>> for (;;) {
>>>> elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); @@
>>>> -452,6 +524,17 @@ static void virtio_iommu_handle_command(VirtIODevice
>> *vdev, VirtQueue *vq)
>>>> case VIRTIO_IOMMU_T_UNMAP:
>>>> tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
>>>> break;
>>>> + case VIRTIO_IOMMU_T_PROBE:
>
> As per spec
> "
> If the device does not offer the VIRTIO_IOMMU_F_PROBE feature, and if the driver sends a VIRTIO_-
> IOMMU_T_PROBE request, then the device SHOULD NOT write the buffer and SHOULD set the used
> length to zero.
> "
> So we should check if device supports "VIRTIO_IOMMU_F_PROBE" before proceed?
But are the device and from that patch onwards we do support the
VIRTIO_IOMMU_F_PROBE feature, right?
Thanks
Eric
>
> Thanks
> -Bharat
>
>>>> + {
>>>> + struct virtio_iommu_req_tail *ptail;
>>>> +
>>>> + output_size = s->config.probe_size + sizeof(tail);
>>>> + buf = g_malloc0(output_size);
>>>> +
>>>> + ptail = (struct virtio_iommu_req_tail *)
>>>> + (buf + s->config.probe_size);
>>>> + ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
>>>> + }
>>>> default:
>>>> tail.status = VIRTIO_IOMMU_S_UNSUPP;
>>>> }
>>>> @@ -459,12 +542,13 @@ static void
>>>> virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
>>>>
>>>> out:
>>>> sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
>>>> - &tail, sizeof(tail));
>>>> - assert(sz == sizeof(tail));
>>>> + buf ? buf : &tail, output_size);
>>>> + assert(sz == output_size);
>>>>
>>>> - virtqueue_push(vq, elem, sizeof(tail));
>>>> + virtqueue_push(vq, elem, sz);
>>>> virtio_notify(vdev, vq);
>>>> g_free(elem);
>>>> + g_free(buf);
>>>> }
>>>> }
>>>>
>>>> @@ -667,6 +751,7 @@ static void
>>>> virtio_iommu_device_realize(DeviceState *dev, Error **errp)
>>>> s->config.page_size_mask = TARGET_PAGE_MASK;
>>>> s->config.input_range.end = -1UL;
>>>> s->config.domain_range.end = 32;
>>>> + s->config.probe_size = VIOMMU_PROBE_SIZE;
>>>>
>>>> virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
>>>> virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC);
>>>> @@ -676,6
>>>> +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev,
>>>> +Error
>>>> **errp)
>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
>>>> + virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
>>>>
>>>> qemu_mutex_init(&s->mutex);
>>>>
>>>> diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index
>>>> e83500bee9..5550475691 100644
>>>> --- a/hw/virtio/trace-events
>>>> +++ b/hw/virtio/trace-events
>>>> @@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc
>>>> domain=%d"
>>>> virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
>>>> virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr,
>>>> uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
>>>> virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t
>>>> endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d
>>>> address =0x%"PRIx64
>>>> +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype,
>>>> +uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64"
>>>> +end=0x%"PRIx64
>>>> --
>>>> 2.20.1
>>>
>>>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 3/5] virtio-iommu: Handle reserved regions in the translation process
2020-05-08 17:30 ` [PATCH v2 3/5] virtio-iommu: Handle reserved regions in the translation process Eric Auger
@ 2020-05-11 21:11 ` Peter Xu
0 siblings, 0 replies; 15+ messages in thread
From: Peter Xu @ 2020-05-11 21:11 UTC (permalink / raw)
To: Eric Auger
Cc: peter.maydell, jean-philippe, mst, qemu-devel, armbru, qemu-arm,
pbonzini, bbhushan2, eric.auger.pro
On Fri, May 08, 2020 at 07:30:55PM +0200, Eric Auger wrote:
> When translating an address we need to check if it belongs to
> a reserved virtual address range. If it does, there are 2 cases:
>
> - it belongs to a RESERVED region: the guest should neither use
> this address in a MAP not instruct the end-point to DMA on
> them. We report an error
>
> - It belongs to an MSI region: we bypass the translation.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
--
Peter Xu
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-11 8:49 ` Auger Eric
@ 2020-05-12 3:03 ` Bharat Bhushan
2020-05-12 3:08 ` Auger Eric
0 siblings, 1 reply; 15+ messages in thread
From: Bharat Bhushan @ 2020-05-12 3:03 UTC (permalink / raw)
To: Auger Eric, eric.auger.pro, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, peterx, armbru, pbonzini
Hi Eric,
> -----Original Message-----
> From: Auger Eric <eric.auger@redhat.com>
> Sent: Monday, May 11, 2020 2:19 PM
> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
> armbru@redhat.com; pbonzini@redhat.com
> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe
> request
>
> Hi Bharat,
>
> On 5/11/20 10:42 AM, Bharat Bhushan wrote:
> > Hi Eric,
> >
> >> -----Original Message-----
> >> From: Auger Eric <eric.auger@redhat.com>
> >> Sent: Monday, May 11, 2020 12:26 PM
> >> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
> >> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
> >> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
> >> armbru@redhat.com; pbonzini@redhat.com
> >> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM
> >> probe request
> >>
> >> Hi Bharat,
> >> On 5/11/20 8:38 AM, Bharat Bhushan wrote:
> >>> Hi Eric,
> >>>
> >>>> -----Original Message-----
> >>>> From: Eric Auger <eric.auger@redhat.com>
> >>>> Sent: Friday, May 8, 2020 11:01 PM
> >>>> To: eric.auger.pro@gmail.com; eric.auger@redhat.com;
> >>>> qemu-devel@nongnu.org; qemu-arm@nongnu.org;
> >>>> peter.maydell@linaro.org; mst@redhat.com; jean-
> >>>> philippe@linaro.org; Bharat Bhushan <bbhushan2@marvell.com>;
> >>>> peterx@redhat.com; armbru@redhat.com; pbonzini@redhat.com
> >>>> Subject: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM
> >>>> probe request
> >>>>
> >>>> External Email
> >>>>
> >>>> -------------------------------------------------------------------
> >>>> --
> >>>> - This patch implements the PROBE request. At the moment, only THE
> >>>> RESV_MEM property is handled. The first goal is to report iommu
> >>>> wide reserved regions such as the MSI regions set by the machine
> >>>> code. On
> >>>> x86 this will be the IOAPIC MSI region,
> >>>> [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
> >>>>
> >>>> In the future we may introduce per device reserved regions.
> >>>> This will be useful when protecting host assigned devices which may
> >>>> expose their own reserved regions
> >>>>
> >>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>
> >>>> ---
> >>>>
> >>>> v1 -> v2:
> >>>> - move the unlock back to the same place
> >>>> - remove the push label and factorize the code after the out label
> >>>> - fix a bunch of cpu_to_leX according to the latest spec revision
> >>>> - do not remove sizeof(last) from free space
> >>>> - check the ep exists
> >>>> ---
> >>>> include/hw/virtio/virtio-iommu.h | 2 +
> >>>> hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
> >>>> hw/virtio/trace-events | 1 +
> >>>> 3 files changed, 93 insertions(+), 4 deletions(-)
> >>>>
> >>>> diff --git a/include/hw/virtio/virtio-iommu.h
> >>>> b/include/hw/virtio/virtio-iommu.h
> >>>> index e653004d7c..49eb105cd8 100644
> >>>> --- a/include/hw/virtio/virtio-iommu.h
> >>>> +++ b/include/hw/virtio/virtio-iommu.h
> >>>> @@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
> >>>> GHashTable *as_by_busptr;
> >>>> IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
> >>>> PCIBus *primary_bus;
> >>>> + ReservedRegion *reserved_regions;
> >>>> + uint32_t nb_reserved_regions;
> >>>> GTree *domains;
> >>>> QemuMutex mutex;
> >>>> GTree *endpoints;
> >>>> diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c
> >>>> index
> >>>> 22ba8848c2..35d772e021 100644
> >>>> --- a/hw/virtio/virtio-iommu.c
> >>>> +++ b/hw/virtio/virtio-iommu.c
> >>>> @@ -38,6 +38,7 @@
> >>>>
> >>>> /* Max size */
> >>>> #define VIOMMU_DEFAULT_QUEUE_SIZE 256
> >>>> +#define VIOMMU_PROBE_SIZE 512
> >>>>
> >>>> typedef struct VirtIOIOMMUDomain {
> >>>> uint32_t id;
> >>>> @@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
> >>>> return ret;
> >>>> }
> >>>>
> >>>> +static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t
> ep,
> >>>> + uint8_t *buf,
> >>>> +size_t
> >>>> +free) {
> >>>> + struct virtio_iommu_probe_resv_mem prop = {};
> >>>> + size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
> >>>> + int i;
> >>>> +
> >>>> + total = size * s->nb_reserved_regions;
> >>>> +
> >>>> + if (total > free) {
> >>>> + return -ENOSPC;
> >>>> + }
> >>>> +
> >>>> + for (i = 0; i < s->nb_reserved_regions; i++) {
> >>>> + prop.head.type = cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
> >>>> + prop.head.length = cpu_to_le16(length);
> >>>> + prop.subtype = s->reserved_regions[i].type;
> >>>> + prop.start = cpu_to_le64(s->reserved_regions[i].low);
> >>>> + prop.end = cpu_to_le64(s->reserved_regions[i].high);
> >>>> +
> >>>> + memcpy(buf, &prop, size);
> >>>> +
> >>>> + trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
> >>>> + prop.start, prop.end);
> >>>> + buf += size;
> >>>> + }
> >>>> + return total;
> >>>> +}
> >>>> +
> >>>> +/**
> >>>> + * virtio_iommu_probe - Fill the probe request buffer with
> >>>> + * the properties the device is able to return and add a NONE
> >>>> + * property at the end.
> >>>> + */
> >>>> +static int virtio_iommu_probe(VirtIOIOMMU *s,
> >>>> + struct virtio_iommu_req_probe *req,
> >>>> + uint8_t *buf) {
> >>>> + uint32_t ep_id = le32_to_cpu(req->endpoint);
> >>>> + size_t free = VIOMMU_PROBE_SIZE;
> >>>> + ssize_t count;
> >>>> +
> >>>> + if (!virtio_iommu_mr(s, ep_id)) {
> >>>> + return VIRTIO_IOMMU_S_NOENT;
> >>>> + }
> >>>> +
> >>>> + count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
> >>>> + if (count < 0) {
> >>>> + return VIRTIO_IOMMU_S_INVAL;
> >>>> + }
> >>>> + buf += count;
> >>>> + free -= count;
> >>>> +
> >>>> + /* Fill the rest with zeroes */
> >>>> + memset(buf, 0, free);
> >>>
> >>> No need to fill with zero here as "buf" is set to zero on allocation, no?
> >>
> >> You're right. I will remove this in the next version.
> >>
> >> Thanks
> >>
> >> Eric
> >>>
> >>> Thanks
> >>> -Bharat
> >>>
> >>>> +
> >>>> + return VIRTIO_IOMMU_S_OK;
> >>>> +}
> >>>> +
> >>>> static int virtio_iommu_iov_to_req(struct iovec *iov,
> >>>> unsigned int iov_cnt,
> >>>> void *req, size_t req_sz) @@
> >>>> -407,15 +467,27 @@
> >>>> virtio_iommu_handle_req(detach)
> >>>> virtio_iommu_handle_req(map)
> >>>> virtio_iommu_handle_req(unmap)
> >>>>
> >>>> +static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
> >>>> + struct iovec *iov,
> >>>> + unsigned int iov_cnt,
> >>>> + uint8_t *buf) {
> >>>> + struct virtio_iommu_req_probe req;
> >>>> + int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req,
> >>>> +sizeof(req));
> >>>> +
> >>>> + return ret ? ret : virtio_iommu_probe(s, &req, buf); }
> >>>> +
> >>>> static void virtio_iommu_handle_command(VirtIODevice *vdev,
> >>>> VirtQueue
> >> *vq) {
> >>>> VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
> >>>> struct virtio_iommu_req_head head;
> >>>> struct virtio_iommu_req_tail tail = {};
> >>>> + size_t output_size = sizeof(tail), sz;
> >>>> VirtQueueElement *elem;
> >>>> unsigned int iov_cnt;
> >>>> struct iovec *iov;
> >>>> - size_t sz;
> >>>> + void *buf = NULL;
> >>>>
> >>>> for (;;) {
> >>>> elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); @@
> >>>> -452,6 +524,17 @@ static void
> >>>> virtio_iommu_handle_command(VirtIODevice
> >> *vdev, VirtQueue *vq)
> >>>> case VIRTIO_IOMMU_T_UNMAP:
> >>>> tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
> >>>> break;
> >>>> + case VIRTIO_IOMMU_T_PROBE:
> >
> > As per spec
> > "
> > If the device does not offer the VIRTIO_IOMMU_F_PROBE feature, and if the
> driver sends a VIRTIO_-
> > IOMMU_T_PROBE request, then the device SHOULD NOT write the buffer and
> SHOULD set the used
> > length to zero.
> > "
> > So we should check if device supports "VIRTIO_IOMMU_F_PROBE" before
> proceed?
> But are the device and from that patch onwards we do support the
> VIRTIO_IOMMU_F_PROBE feature, right?
Yes I agree, do you think if for debugging one wants to try out without this feature then he should just disable VIRTIO_IOMMU_F_PROBE.
Thanks
-Bharat
>
> Thanks
>
> Eric
> >
> > Thanks
> > -Bharat
> >
> >>>> + {
> >>>> + struct virtio_iommu_req_tail *ptail;
> >>>> +
> >>>> + output_size = s->config.probe_size + sizeof(tail);
> >>>> + buf = g_malloc0(output_size);
> >>>> +
> >>>> + ptail = (struct virtio_iommu_req_tail *)
> >>>> + (buf + s->config.probe_size);
> >>>> + ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
> >>>> + }
> >>>> default:
> >>>> tail.status = VIRTIO_IOMMU_S_UNSUPP;
> >>>> }
> >>>> @@ -459,12 +542,13 @@ static void
> >>>> virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
> >>>>
> >>>> out:
> >>>> sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
> >>>> - &tail, sizeof(tail));
> >>>> - assert(sz == sizeof(tail));
> >>>> + buf ? buf : &tail, output_size);
> >>>> + assert(sz == output_size);
> >>>>
> >>>> - virtqueue_push(vq, elem, sizeof(tail));
> >>>> + virtqueue_push(vq, elem, sz);
> >>>> virtio_notify(vdev, vq);
> >>>> g_free(elem);
> >>>> + g_free(buf);
> >>>> }
> >>>> }
> >>>>
> >>>> @@ -667,6 +751,7 @@ static void
> >>>> virtio_iommu_device_realize(DeviceState *dev, Error **errp)
> >>>> s->config.page_size_mask = TARGET_PAGE_MASK;
> >>>> s->config.input_range.end = -1UL;
> >>>> s->config.domain_range.end = 32;
> >>>> + s->config.probe_size = VIOMMU_PROBE_SIZE;
> >>>>
> >>>> virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
> >>>> virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC);
> >>>> @@ -676,6
> >>>> +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev,
> >>>> +Error
> >>>> **errp)
> >>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
> >>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
> >>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
> >>>> + virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
> >>>>
> >>>> qemu_mutex_init(&s->mutex);
> >>>>
> >>>> diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index
> >>>> e83500bee9..5550475691 100644
> >>>> --- a/hw/virtio/trace-events
> >>>> +++ b/hw/virtio/trace-events
> >>>> @@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id)
> >>>> "Alloc domain=%d"
> >>>> virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
> >>>> virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr,
> >>>> uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
> >>>> virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t
> >>>> endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d
> >>>> address =0x%"PRIx64
> >>>> +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype,
> >>>> +uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64"
> >>>> +end=0x%"PRIx64
> >>>> --
> >>>> 2.20.1
> >>>
> >>>
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-12 3:03 ` Bharat Bhushan
@ 2020-05-12 3:08 ` Auger Eric
2020-05-12 3:11 ` Bharat Bhushan
0 siblings, 1 reply; 15+ messages in thread
From: Auger Eric @ 2020-05-12 3:08 UTC (permalink / raw)
To: Bharat Bhushan, eric.auger.pro, qemu-devel, qemu-arm,
peter.maydell, mst, jean-philippe, peterx, armbru, pbonzini
Hi Bharat,
On 5/12/20 5:03 AM, Bharat Bhushan wrote:
> Hi Eric,
>
>> -----Original Message-----
>> From: Auger Eric <eric.auger@redhat.com>
>> Sent: Monday, May 11, 2020 2:19 PM
>> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
>> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
>> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
>> armbru@redhat.com; pbonzini@redhat.com
>> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe
>> request
>>
>> Hi Bharat,
>>
>> On 5/11/20 10:42 AM, Bharat Bhushan wrote:
>>> Hi Eric,
>>>
>>>> -----Original Message-----
>>>> From: Auger Eric <eric.auger@redhat.com>
>>>> Sent: Monday, May 11, 2020 12:26 PM
>>>> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
>>>> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
>>>> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
>>>> armbru@redhat.com; pbonzini@redhat.com
>>>> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM
>>>> probe request
>>>>
>>>> Hi Bharat,
>>>> On 5/11/20 8:38 AM, Bharat Bhushan wrote:
>>>>> Hi Eric,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Eric Auger <eric.auger@redhat.com>
>>>>>> Sent: Friday, May 8, 2020 11:01 PM
>>>>>> To: eric.auger.pro@gmail.com; eric.auger@redhat.com;
>>>>>> qemu-devel@nongnu.org; qemu-arm@nongnu.org;
>>>>>> peter.maydell@linaro.org; mst@redhat.com; jean-
>>>>>> philippe@linaro.org; Bharat Bhushan <bbhushan2@marvell.com>;
>>>>>> peterx@redhat.com; armbru@redhat.com; pbonzini@redhat.com
>>>>>> Subject: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM
>>>>>> probe request
>>>>>>
>>>>>> External Email
>>>>>>
>>>>>> -------------------------------------------------------------------
>>>>>> --
>>>>>> - This patch implements the PROBE request. At the moment, only THE
>>>>>> RESV_MEM property is handled. The first goal is to report iommu
>>>>>> wide reserved regions such as the MSI regions set by the machine
>>>>>> code. On
>>>>>> x86 this will be the IOAPIC MSI region,
>>>>>> [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
>>>>>>
>>>>>> In the future we may introduce per device reserved regions.
>>>>>> This will be useful when protecting host assigned devices which may
>>>>>> expose their own reserved regions
>>>>>>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> v1 -> v2:
>>>>>> - move the unlock back to the same place
>>>>>> - remove the push label and factorize the code after the out label
>>>>>> - fix a bunch of cpu_to_leX according to the latest spec revision
>>>>>> - do not remove sizeof(last) from free space
>>>>>> - check the ep exists
>>>>>> ---
>>>>>> include/hw/virtio/virtio-iommu.h | 2 +
>>>>>> hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
>>>>>> hw/virtio/trace-events | 1 +
>>>>>> 3 files changed, 93 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/include/hw/virtio/virtio-iommu.h
>>>>>> b/include/hw/virtio/virtio-iommu.h
>>>>>> index e653004d7c..49eb105cd8 100644
>>>>>> --- a/include/hw/virtio/virtio-iommu.h
>>>>>> +++ b/include/hw/virtio/virtio-iommu.h
>>>>>> @@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
>>>>>> GHashTable *as_by_busptr;
>>>>>> IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
>>>>>> PCIBus *primary_bus;
>>>>>> + ReservedRegion *reserved_regions;
>>>>>> + uint32_t nb_reserved_regions;
>>>>>> GTree *domains;
>>>>>> QemuMutex mutex;
>>>>>> GTree *endpoints;
>>>>>> diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c
>>>>>> index
>>>>>> 22ba8848c2..35d772e021 100644
>>>>>> --- a/hw/virtio/virtio-iommu.c
>>>>>> +++ b/hw/virtio/virtio-iommu.c
>>>>>> @@ -38,6 +38,7 @@
>>>>>>
>>>>>> /* Max size */
>>>>>> #define VIOMMU_DEFAULT_QUEUE_SIZE 256
>>>>>> +#define VIOMMU_PROBE_SIZE 512
>>>>>>
>>>>>> typedef struct VirtIOIOMMUDomain {
>>>>>> uint32_t id;
>>>>>> @@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
>>>>>> return ret;
>>>>>> }
>>>>>>
>>>>>> +static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t
>> ep,
>>>>>> + uint8_t *buf,
>>>>>> +size_t
>>>>>> +free) {
>>>>>> + struct virtio_iommu_probe_resv_mem prop = {};
>>>>>> + size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
>>>>>> + int i;
>>>>>> +
>>>>>> + total = size * s->nb_reserved_regions;
>>>>>> +
>>>>>> + if (total > free) {
>>>>>> + return -ENOSPC;
>>>>>> + }
>>>>>> +
>>>>>> + for (i = 0; i < s->nb_reserved_regions; i++) {
>>>>>> + prop.head.type = cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
>>>>>> + prop.head.length = cpu_to_le16(length);
>>>>>> + prop.subtype = s->reserved_regions[i].type;
>>>>>> + prop.start = cpu_to_le64(s->reserved_regions[i].low);
>>>>>> + prop.end = cpu_to_le64(s->reserved_regions[i].high);
>>>>>> +
>>>>>> + memcpy(buf, &prop, size);
>>>>>> +
>>>>>> + trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
>>>>>> + prop.start, prop.end);
>>>>>> + buf += size;
>>>>>> + }
>>>>>> + return total;
>>>>>> +}
>>>>>> +
>>>>>> +/**
>>>>>> + * virtio_iommu_probe - Fill the probe request buffer with
>>>>>> + * the properties the device is able to return and add a NONE
>>>>>> + * property at the end.
>>>>>> + */
>>>>>> +static int virtio_iommu_probe(VirtIOIOMMU *s,
>>>>>> + struct virtio_iommu_req_probe *req,
>>>>>> + uint8_t *buf) {
>>>>>> + uint32_t ep_id = le32_to_cpu(req->endpoint);
>>>>>> + size_t free = VIOMMU_PROBE_SIZE;
>>>>>> + ssize_t count;
>>>>>> +
>>>>>> + if (!virtio_iommu_mr(s, ep_id)) {
>>>>>> + return VIRTIO_IOMMU_S_NOENT;
>>>>>> + }
>>>>>> +
>>>>>> + count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
>>>>>> + if (count < 0) {
>>>>>> + return VIRTIO_IOMMU_S_INVAL;
>>>>>> + }
>>>>>> + buf += count;
>>>>>> + free -= count;
>>>>>> +
>>>>>> + /* Fill the rest with zeroes */
>>>>>> + memset(buf, 0, free);
>>>>>
>>>>> No need to fill with zero here as "buf" is set to zero on allocation, no?
>>>>
>>>> You're right. I will remove this in the next version.
>>>>
>>>> Thanks
>>>>
>>>> Eric
>>>>>
>>>>> Thanks
>>>>> -Bharat
>>>>>
>>>>>> +
>>>>>> + return VIRTIO_IOMMU_S_OK;
>>>>>> +}
>>>>>> +
>>>>>> static int virtio_iommu_iov_to_req(struct iovec *iov,
>>>>>> unsigned int iov_cnt,
>>>>>> void *req, size_t req_sz) @@
>>>>>> -407,15 +467,27 @@
>>>>>> virtio_iommu_handle_req(detach)
>>>>>> virtio_iommu_handle_req(map)
>>>>>> virtio_iommu_handle_req(unmap)
>>>>>>
>>>>>> +static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
>>>>>> + struct iovec *iov,
>>>>>> + unsigned int iov_cnt,
>>>>>> + uint8_t *buf) {
>>>>>> + struct virtio_iommu_req_probe req;
>>>>>> + int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req,
>>>>>> +sizeof(req));
>>>>>> +
>>>>>> + return ret ? ret : virtio_iommu_probe(s, &req, buf); }
>>>>>> +
>>>>>> static void virtio_iommu_handle_command(VirtIODevice *vdev,
>>>>>> VirtQueue
>>>> *vq) {
>>>>>> VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
>>>>>> struct virtio_iommu_req_head head;
>>>>>> struct virtio_iommu_req_tail tail = {};
>>>>>> + size_t output_size = sizeof(tail), sz;
>>>>>> VirtQueueElement *elem;
>>>>>> unsigned int iov_cnt;
>>>>>> struct iovec *iov;
>>>>>> - size_t sz;
>>>>>> + void *buf = NULL;
>>>>>>
>>>>>> for (;;) {
>>>>>> elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); @@
>>>>>> -452,6 +524,17 @@ static void
>>>>>> virtio_iommu_handle_command(VirtIODevice
>>>> *vdev, VirtQueue *vq)
>>>>>> case VIRTIO_IOMMU_T_UNMAP:
>>>>>> tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
>>>>>> break;
>>>>>> + case VIRTIO_IOMMU_T_PROBE:
>>>
>>> As per spec
>>> "
>>> If the device does not offer the VIRTIO_IOMMU_F_PROBE feature, and if the
>> driver sends a VIRTIO_-
>>> IOMMU_T_PROBE request, then the device SHOULD NOT write the buffer and
>> SHOULD set the used
>>> length to zero.
>>> "
>>> So we should check if device supports "VIRTIO_IOMMU_F_PROBE" before
>> proceed?
>> But are the device and from that patch onwards we do support the
>> VIRTIO_IOMMU_F_PROBE feature, right?
>
> Yes I agree, do you think if for debugging one wants to try out without this feature then he should just disable VIRTIO_IOMMU_F_PROBE.
You mean for debugging the driver? I don't think this is the purpose of
this device.
Thanks
Eric
>
> Thanks
> -Bharat
>
>>
>> Thanks
>>
>> Eric
>>>
>>> Thanks
>>> -Bharat
>>>
>>>>>> + {
>>>>>> + struct virtio_iommu_req_tail *ptail;
>>>>>> +
>>>>>> + output_size = s->config.probe_size + sizeof(tail);
>>>>>> + buf = g_malloc0(output_size);
>>>>>> +
>>>>>> + ptail = (struct virtio_iommu_req_tail *)
>>>>>> + (buf + s->config.probe_size);
>>>>>> + ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
>>>>>> + }
>>>>>> default:
>>>>>> tail.status = VIRTIO_IOMMU_S_UNSUPP;
>>>>>> }
>>>>>> @@ -459,12 +542,13 @@ static void
>>>>>> virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
>>>>>>
>>>>>> out:
>>>>>> sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
>>>>>> - &tail, sizeof(tail));
>>>>>> - assert(sz == sizeof(tail));
>>>>>> + buf ? buf : &tail, output_size);
>>>>>> + assert(sz == output_size);
>>>>>>
>>>>>> - virtqueue_push(vq, elem, sizeof(tail));
>>>>>> + virtqueue_push(vq, elem, sz);
>>>>>> virtio_notify(vdev, vq);
>>>>>> g_free(elem);
>>>>>> + g_free(buf);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> @@ -667,6 +751,7 @@ static void
>>>>>> virtio_iommu_device_realize(DeviceState *dev, Error **errp)
>>>>>> s->config.page_size_mask = TARGET_PAGE_MASK;
>>>>>> s->config.input_range.end = -1UL;
>>>>>> s->config.domain_range.end = 32;
>>>>>> + s->config.probe_size = VIOMMU_PROBE_SIZE;
>>>>>>
>>>>>> virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
>>>>>> virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC);
>>>>>> @@ -676,6
>>>>>> +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev,
>>>>>> +Error
>>>>>> **errp)
>>>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
>>>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
>>>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
>>>>>> + virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
>>>>>>
>>>>>> qemu_mutex_init(&s->mutex);
>>>>>>
>>>>>> diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index
>>>>>> e83500bee9..5550475691 100644
>>>>>> --- a/hw/virtio/trace-events
>>>>>> +++ b/hw/virtio/trace-events
>>>>>> @@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id)
>>>>>> "Alloc domain=%d"
>>>>>> virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
>>>>>> virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr,
>>>>>> uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
>>>>>> virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t
>>>>>> endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d
>>>>>> address =0x%"PRIx64
>>>>>> +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype,
>>>>>> +uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64"
>>>>>> +end=0x%"PRIx64
>>>>>> --
>>>>>> 2.20.1
>>>>>
>>>>>
>>>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request
2020-05-12 3:08 ` Auger Eric
@ 2020-05-12 3:11 ` Bharat Bhushan
0 siblings, 0 replies; 15+ messages in thread
From: Bharat Bhushan @ 2020-05-12 3:11 UTC (permalink / raw)
To: Auger Eric, eric.auger.pro, qemu-devel, qemu-arm, peter.maydell,
mst, jean-philippe, peterx, armbru, pbonzini
Hi Eric,
> -----Original Message-----
> From: Auger Eric <eric.auger@redhat.com>
> Sent: Tuesday, May 12, 2020 8:39 AM
> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
> armbru@redhat.com; pbonzini@redhat.com
> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe
> request
>
> Hi Bharat,
> On 5/12/20 5:03 AM, Bharat Bhushan wrote:
> > Hi Eric,
> >
> >> -----Original Message-----
> >> From: Auger Eric <eric.auger@redhat.com>
> >> Sent: Monday, May 11, 2020 2:19 PM
> >> To: Bharat Bhushan <bbhushan2@marvell.com>; eric.auger.pro@gmail.com;
> >> qemu-devel@nongnu.org; qemu-arm@nongnu.org; peter.maydell@linaro.org;
> >> mst@redhat.com; jean-philippe@linaro.org; peterx@redhat.com;
> >> armbru@redhat.com; pbonzini@redhat.com
> >> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM
> >> probe request
> >>
> >> Hi Bharat,
> >>
> >> On 5/11/20 10:42 AM, Bharat Bhushan wrote:
> >>> Hi Eric,
> >>>
> >>>> -----Original Message-----
> >>>> From: Auger Eric <eric.auger@redhat.com>
> >>>> Sent: Monday, May 11, 2020 12:26 PM
> >>>> To: Bharat Bhushan <bbhushan2@marvell.com>;
> >>>> eric.auger.pro@gmail.com; qemu-devel@nongnu.org;
> >>>> qemu-arm@nongnu.org; peter.maydell@linaro.org; mst@redhat.com;
> >>>> jean-philippe@linaro.org; peterx@redhat.com; armbru@redhat.com;
> >>>> pbonzini@redhat.com
> >>>> Subject: Re: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM
> >>>> probe request
> >>>>
> >>>> Hi Bharat,
> >>>> On 5/11/20 8:38 AM, Bharat Bhushan wrote:
> >>>>> Hi Eric,
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Eric Auger <eric.auger@redhat.com>
> >>>>>> Sent: Friday, May 8, 2020 11:01 PM
> >>>>>> To: eric.auger.pro@gmail.com; eric.auger@redhat.com;
> >>>>>> qemu-devel@nongnu.org; qemu-arm@nongnu.org;
> >>>>>> peter.maydell@linaro.org; mst@redhat.com; jean-
> >>>>>> philippe@linaro.org; Bharat Bhushan <bbhushan2@marvell.com>;
> >>>>>> peterx@redhat.com; armbru@redhat.com; pbonzini@redhat.com
> >>>>>> Subject: [EXT] [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM
> >>>>>> probe request
> >>>>>>
> >>>>>> External Email
> >>>>>>
> >>>>>> -----------------------------------------------------------------
> >>>>>> --
> >>>>>> --
> >>>>>> - This patch implements the PROBE request. At the moment, only
> >>>>>> THE RESV_MEM property is handled. The first goal is to report
> >>>>>> iommu wide reserved regions such as the MSI regions set by the
> >>>>>> machine code. On
> >>>>>> x86 this will be the IOAPIC MSI region,
> >>>>>> [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
> >>>>>>
> >>>>>> In the future we may introduce per device reserved regions.
> >>>>>> This will be useful when protecting host assigned devices which
> >>>>>> may expose their own reserved regions
> >>>>>>
> >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>
> >>>>>> ---
> >>>>>>
> >>>>>> v1 -> v2:
> >>>>>> - move the unlock back to the same place
> >>>>>> - remove the push label and factorize the code after the out
> >>>>>> label
> >>>>>> - fix a bunch of cpu_to_leX according to the latest spec revision
> >>>>>> - do not remove sizeof(last) from free space
> >>>>>> - check the ep exists
> >>>>>> ---
> >>>>>> include/hw/virtio/virtio-iommu.h | 2 +
> >>>>>> hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++--
> >>>>>> hw/virtio/trace-events | 1 +
> >>>>>> 3 files changed, 93 insertions(+), 4 deletions(-)
> >>>>>>
> >>>>>> diff --git a/include/hw/virtio/virtio-iommu.h
> >>>>>> b/include/hw/virtio/virtio-iommu.h
> >>>>>> index e653004d7c..49eb105cd8 100644
> >>>>>> --- a/include/hw/virtio/virtio-iommu.h
> >>>>>> +++ b/include/hw/virtio/virtio-iommu.h
> >>>>>> @@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
> >>>>>> GHashTable *as_by_busptr;
> >>>>>> IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
> >>>>>> PCIBus *primary_bus;
> >>>>>> + ReservedRegion *reserved_regions;
> >>>>>> + uint32_t nb_reserved_regions;
> >>>>>> GTree *domains;
> >>>>>> QemuMutex mutex;
> >>>>>> GTree *endpoints;
> >>>>>> diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c
> >>>>>> index
> >>>>>> 22ba8848c2..35d772e021 100644
> >>>>>> --- a/hw/virtio/virtio-iommu.c
> >>>>>> +++ b/hw/virtio/virtio-iommu.c
> >>>>>> @@ -38,6 +38,7 @@
> >>>>>>
> >>>>>> /* Max size */
> >>>>>> #define VIOMMU_DEFAULT_QUEUE_SIZE 256
> >>>>>> +#define VIOMMU_PROBE_SIZE 512
> >>>>>>
> >>>>>> typedef struct VirtIOIOMMUDomain {
> >>>>>> uint32_t id;
> >>>>>> @@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,
> >>>>>> return ret;
> >>>>>> }
> >>>>>>
> >>>>>> +static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s,
> >>>>>> +uint32_t
> >> ep,
> >>>>>> + uint8_t *buf,
> >>>>>> +size_t
> >>>>>> +free) {
> >>>>>> + struct virtio_iommu_probe_resv_mem prop = {};
> >>>>>> + size_t size = sizeof(prop), length = size - sizeof(prop.head), total;
> >>>>>> + int i;
> >>>>>> +
> >>>>>> + total = size * s->nb_reserved_regions;
> >>>>>> +
> >>>>>> + if (total > free) {
> >>>>>> + return -ENOSPC;
> >>>>>> + }
> >>>>>> +
> >>>>>> + for (i = 0; i < s->nb_reserved_regions; i++) {
> >>>>>> + prop.head.type =
> cpu_to_le16(VIRTIO_IOMMU_PROBE_T_RESV_MEM);
> >>>>>> + prop.head.length = cpu_to_le16(length);
> >>>>>> + prop.subtype = s->reserved_regions[i].type;
> >>>>>> + prop.start = cpu_to_le64(s->reserved_regions[i].low);
> >>>>>> + prop.end = cpu_to_le64(s->reserved_regions[i].high);
> >>>>>> +
> >>>>>> + memcpy(buf, &prop, size);
> >>>>>> +
> >>>>>> + trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
> >>>>>> + prop.start, prop.end);
> >>>>>> + buf += size;
> >>>>>> + }
> >>>>>> + return total;
> >>>>>> +}
> >>>>>> +
> >>>>>> +/**
> >>>>>> + * virtio_iommu_probe - Fill the probe request buffer with
> >>>>>> + * the properties the device is able to return and add a NONE
> >>>>>> + * property at the end.
> >>>>>> + */
> >>>>>> +static int virtio_iommu_probe(VirtIOIOMMU *s,
> >>>>>> + struct virtio_iommu_req_probe *req,
> >>>>>> + uint8_t *buf) {
> >>>>>> + uint32_t ep_id = le32_to_cpu(req->endpoint);
> >>>>>> + size_t free = VIOMMU_PROBE_SIZE;
> >>>>>> + ssize_t count;
> >>>>>> +
> >>>>>> + if (!virtio_iommu_mr(s, ep_id)) {
> >>>>>> + return VIRTIO_IOMMU_S_NOENT;
> >>>>>> + }
> >>>>>> +
> >>>>>> + count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
> >>>>>> + if (count < 0) {
> >>>>>> + return VIRTIO_IOMMU_S_INVAL;
> >>>>>> + }
> >>>>>> + buf += count;
> >>>>>> + free -= count;
> >>>>>> +
> >>>>>> + /* Fill the rest with zeroes */
> >>>>>> + memset(buf, 0, free);
> >>>>>
> >>>>> No need to fill with zero here as "buf" is set to zero on allocation, no?
> >>>>
> >>>> You're right. I will remove this in the next version.
> >>>>
> >>>> Thanks
> >>>>
> >>>> Eric
> >>>>>
> >>>>> Thanks
> >>>>> -Bharat
> >>>>>
> >>>>>> +
> >>>>>> + return VIRTIO_IOMMU_S_OK;
> >>>>>> +}
> >>>>>> +
> >>>>>> static int virtio_iommu_iov_to_req(struct iovec *iov,
> >>>>>> unsigned int iov_cnt,
> >>>>>> void *req, size_t req_sz) @@
> >>>>>> -407,15 +467,27 @@
> >>>>>> virtio_iommu_handle_req(detach)
> >>>>>> virtio_iommu_handle_req(map)
> >>>>>> virtio_iommu_handle_req(unmap)
> >>>>>>
> >>>>>> +static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
> >>>>>> + struct iovec *iov,
> >>>>>> + unsigned int iov_cnt,
> >>>>>> + uint8_t *buf) {
> >>>>>> + struct virtio_iommu_req_probe req;
> >>>>>> + int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req,
> >>>>>> +sizeof(req));
> >>>>>> +
> >>>>>> + return ret ? ret : virtio_iommu_probe(s, &req, buf); }
> >>>>>> +
> >>>>>> static void virtio_iommu_handle_command(VirtIODevice *vdev,
> >>>>>> VirtQueue
> >>>> *vq) {
> >>>>>> VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
> >>>>>> struct virtio_iommu_req_head head;
> >>>>>> struct virtio_iommu_req_tail tail = {};
> >>>>>> + size_t output_size = sizeof(tail), sz;
> >>>>>> VirtQueueElement *elem;
> >>>>>> unsigned int iov_cnt;
> >>>>>> struct iovec *iov;
> >>>>>> - size_t sz;
> >>>>>> + void *buf = NULL;
> >>>>>>
> >>>>>> for (;;) {
> >>>>>> elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); @@
> >>>>>> -452,6 +524,17 @@ static void
> >>>>>> virtio_iommu_handle_command(VirtIODevice
> >>>> *vdev, VirtQueue *vq)
> >>>>>> case VIRTIO_IOMMU_T_UNMAP:
> >>>>>> tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
> >>>>>> break;
> >>>>>> + case VIRTIO_IOMMU_T_PROBE:
> >>>
> >>> As per spec
> >>> "
> >>> If the device does not offer the VIRTIO_IOMMU_F_PROBE feature,
> >>> and if the
> >> driver sends a VIRTIO_-
> >>> IOMMU_T_PROBE request, then the device SHOULD NOT write the
> >>> buffer and
> >> SHOULD set the used
> >>> length to zero.
> >>> "
> >>> So we should check if device supports "VIRTIO_IOMMU_F_PROBE" before
> >> proceed?
> >> But are the device and from that patch onwards we do support the
> >> VIRTIO_IOMMU_F_PROBE feature, right?
> >
> > Yes I agree, do you think if for debugging one wants to try out without this
> feature then he should just disable VIRTIO_IOMMU_F_PROBE.
>
> You mean for debugging the driver? I don't think this is the purpose of this device.
Yes for debugging purpose only.
Thanks
-Bharat
>
> Thanks
>
> Eric
> >
> > Thanks
> > -Bharat
> >
> >>
> >> Thanks
> >>
> >> Eric
> >>>
> >>> Thanks
> >>> -Bharat
> >>>
> >>>>>> + {
> >>>>>> + struct virtio_iommu_req_tail *ptail;
> >>>>>> +
> >>>>>> + output_size = s->config.probe_size + sizeof(tail);
> >>>>>> + buf = g_malloc0(output_size);
> >>>>>> +
> >>>>>> + ptail = (struct virtio_iommu_req_tail *)
> >>>>>> + (buf + s->config.probe_size);
> >>>>>> + ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
> >>>>>> + }
> >>>>>> default:
> >>>>>> tail.status = VIRTIO_IOMMU_S_UNSUPP;
> >>>>>> }
> >>>>>> @@ -459,12 +542,13 @@ static void
> >>>>>> virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
> >>>>>>
> >>>>>> out:
> >>>>>> sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
> >>>>>> - &tail, sizeof(tail));
> >>>>>> - assert(sz == sizeof(tail));
> >>>>>> + buf ? buf : &tail, output_size);
> >>>>>> + assert(sz == output_size);
> >>>>>>
> >>>>>> - virtqueue_push(vq, elem, sizeof(tail));
> >>>>>> + virtqueue_push(vq, elem, sz);
> >>>>>> virtio_notify(vdev, vq);
> >>>>>> g_free(elem);
> >>>>>> + g_free(buf);
> >>>>>> }
> >>>>>> }
> >>>>>>
> >>>>>> @@ -667,6 +751,7 @@ static void
> >>>>>> virtio_iommu_device_realize(DeviceState *dev, Error **errp)
> >>>>>> s->config.page_size_mask = TARGET_PAGE_MASK;
> >>>>>> s->config.input_range.end = -1UL;
> >>>>>> s->config.domain_range.end = 32;
> >>>>>> + s->config.probe_size = VIOMMU_PROBE_SIZE;
> >>>>>>
> >>>>>> virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
> >>>>>> virtio_add_feature(&s->features,
> >>>>>> VIRTIO_RING_F_INDIRECT_DESC); @@ -676,6
> >>>>>> +761,7 @@ static void virtio_iommu_device_realize(DeviceState
> >>>>>> +*dev, Error
> >>>>>> **errp)
> >>>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
> >>>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
> >>>>>> virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
> >>>>>> + virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
> >>>>>>
> >>>>>> qemu_mutex_init(&s->mutex);
> >>>>>>
> >>>>>> diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
> >>>>>> index
> >>>>>> e83500bee9..5550475691 100644
> >>>>>> --- a/hw/virtio/trace-events
> >>>>>> +++ b/hw/virtio/trace-events
> >>>>>> @@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id)
> >>>>>> "Alloc domain=%d"
> >>>>>> virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
> >>>>>> virtio_iommu_translate_out(uint64_t virt_addr, uint64_t
> >>>>>> phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
> >>>>>> virtio_iommu_report_fault(uint8_t reason, uint32_t flags,
> >>>>>> uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d
> >>>>>> endpoint=%d address =0x%"PRIx64
> >>>>>> +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype,
> >>>>>> +uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64"
> >>>>>> +end=0x%"PRIx64
> >>>>>> --
> >>>>>> 2.20.1
> >>>>>
> >>>>>
> >>>
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs
2020-05-08 17:30 ` [PATCH v2 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs Eric Auger
@ 2020-05-22 14:43 ` Jean-Philippe Brucker
0 siblings, 0 replies; 15+ messages in thread
From: Jean-Philippe Brucker @ 2020-05-22 14:43 UTC (permalink / raw)
To: Eric Auger
Cc: peter.maydell, mst, qemu-devel, peterx, armbru, qemu-arm,
pbonzini, bbhushan2, eric.auger.pro
On Fri, May 08, 2020 at 07:30:57PM +0200, Eric Auger wrote:
> At the moment the virtio-iommu translates MSI transactions.
> This behavior is inherited from ARM SMMU. The virt machine
> code knows where the guest MSI doorbells are so we can easily
> declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that
> setting the guest will not map MSIs through the IOMMU and those
> transactions will be simply bypassed.
>
> Depending on which MSI controller is in use (ITS or GICV2M),
> we declare either:
> - the ITS interrupt translation space (ITS_base + 0x10000),
> containing the GITS_TRANSLATOR or
> - The GICV2M single frame, containing the MSI_SETSP_NS register.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
>
> v1 -> v2:
> - Test which MSI controller is instantiated
> - If GICV2M is in use, declare its doorbell as an MSI doorbell too
> ---
> include/hw/arm/virt.h | 6 ++++++
> hw/arm/virt.c | 18 ++++++++++++++++++
> 2 files changed, 24 insertions(+)
>
> diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
> index 6d67ace76e..ad20cb6e15 100644
> --- a/include/hw/arm/virt.h
> +++ b/include/hw/arm/virt.h
> @@ -96,6 +96,11 @@ typedef enum VirtIOMMUType {
> VIRT_IOMMU_VIRTIO,
> } VirtIOMMUType;
>
> +typedef enum VirtMSIControllerType {
> + VIRT_GICV2M,
> + VIRT_ITS,
> +} VirtMSIControllerType;
I think you need a third value for msi_controller == 0. If I instantiate a
GICv3 without ITS at the moment the V2M region gets reserved. Not a big
deal since MSIs aren't supported at all in this case, but it would be
cleaner to skip any reservation.
Thanks,
Jean
> +
> typedef enum VirtGICType {
> VIRT_GIC_VERSION_MAX,
> VIRT_GIC_VERSION_HOST,
> @@ -135,6 +140,7 @@ typedef struct {
> OnOffAuto acpi;
> VirtGICType gic_version;
> VirtIOMMUType iommu;
> + VirtMSIControllerType msi_controller;
> uint16_t virtio_iommu_bdf;
> struct arm_boot_info bootinfo;
> MemMapEntry *memmap;
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 634db0cfe9..d2dd07885b 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -602,6 +602,7 @@ static void create_its(VirtMachineState *vms)
> sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, vms->memmap[VIRT_GIC_ITS].base);
>
> fdt_add_its_gic_node(vms);
> + vms->msi_controller = VIRT_ITS;
> }
>
> static void create_v2m(VirtMachineState *vms)
> @@ -622,6 +623,7 @@ static void create_v2m(VirtMachineState *vms)
> }
>
> fdt_add_v2m_gic_node(vms);
> + vms->msi_controller = VIRT_GICV2M;
> }
>
> static void create_gic(VirtMachineState *vms)
> @@ -2136,8 +2138,24 @@ out:
> static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
> DeviceState *dev, Error **errp)
> {
> + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
> +
> if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
> virt_memory_pre_plug(hotplug_dev, dev, errp);
> + } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
> + /* we declare a VIRTIO_IOMMU_RESV_MEM_T_MSI region */
> +
> + if (vms->msi_controller == VIRT_ITS) {
> + /* GITS_TRANSLATER page */
> + qdev_prop_set_uint32(dev, "len-reserved-regions", 1);
> + qdev_prop_set_string(dev, "reserved-regions[0]",
> + "0x8090000, 0x809FFFF, 1");
> + } else if (vms->msi_controller == VIRT_GICV2M) {
> + /* MSI_SETSPI_NS page */
> + qdev_prop_set_uint32(dev, "len-reserved-regions", 1);
> + qdev_prop_set_string(dev, "reserved-regions[0]",
> + "0x8020000, 0x8020FFF, 1");
> + }
> }
> }
>
> --
> 2.20.1
>
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2020-05-22 14:44 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-08 17:30 [PATCH v2 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM Eric Auger
2020-05-08 17:30 ` [PATCH v2 1/5] qdev: Introduce DEFINE_PROP_RESERVED_REGION Eric Auger
2020-05-08 17:30 ` [PATCH v2 2/5] virtio-iommu: Implement RESV_MEM probe request Eric Auger
2020-05-11 6:38 ` [EXT] " Bharat Bhushan
2020-05-11 6:55 ` Auger Eric
2020-05-11 8:42 ` Bharat Bhushan
2020-05-11 8:49 ` Auger Eric
2020-05-12 3:03 ` Bharat Bhushan
2020-05-12 3:08 ` Auger Eric
2020-05-12 3:11 ` Bharat Bhushan
2020-05-08 17:30 ` [PATCH v2 3/5] virtio-iommu: Handle reserved regions in the translation process Eric Auger
2020-05-11 21:11 ` Peter Xu
2020-05-08 17:30 ` [PATCH v2 4/5] virtio-iommu-pci: Add array of Interval properties Eric Auger
2020-05-08 17:30 ` [PATCH v2 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs Eric Auger
2020-05-22 14:43 ` Jean-Philippe Brucker
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.