[PATCH v3 0/9] kvmtool: handle guests of a different endianness

[PATCH v3 0/9] kvmtool: handle guests of a different endianness

[Marc Zyngier]

This patch series adds some infrastructure to kvmtool to allow a BE
guest to use virtio-mmio on a LE host, provided that the architecture
actually supports such madness.

Not all the backend have been converted, only those I actually cared
about. Converting them is pretty easy though, and will be done if the
method is deemed acceptable.

This has been tested on both arm and arm64 (I use this on a daily
basis to test BE code). The corresponding kernel changes have all been
merged.

Also available at:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/kvmtool-be-on-le

>From v2 (never posted):
- Fixed tons of bugs (config space)
- Fixed TAP networking

>From v1:
- Gave up on the virtio extension after the push back from the PPC
  guys. Instead, we snapshot the endianness of the vcpu  when it
  tries to reset the device. A bit ugly, but doesn't require any
  change on the kernel side.

Marc Zyngier (9):
  kvmtool: pass trapped vcpu to MMIO accessors
  kvmtool: virt_queue configuration based on endianness
  kvmtool: sample CPU endianness on virtio-mmio device reset
  kvmtool: add queue endianness initializer
  kvmtool: convert console backend to support bi-endianness
  kvmtool: convert 9p backend to support bi-endianness
  kvmtool: convert blk backend to support bi-endianness
  kvmtool: convert net backend to support bi-endianness
  kvmtool: virtio: enable arm/arm64 support for bi-endianness

 tools/kvm/arm/aarch32/kvm-cpu.c                  | 14 ++++
 tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h |  2 +
 tools/kvm/arm/aarch64/kvm-cpu.c                  | 25 ++++++++
 tools/kvm/arm/include/arm-common/kvm-arch.h      |  2 +
 tools/kvm/arm/include/arm-common/kvm-cpu-arch.h  |  4 +-
 tools/kvm/arm/kvm-cpu.c                          | 10 +--
 tools/kvm/hw/pci-shmem.c                         |  2 +-
 tools/kvm/include/kvm/kvm-cpu.h                  |  1 +
 tools/kvm/include/kvm/kvm.h                      |  4 +-
 tools/kvm/include/kvm/virtio.h                   | 82 +++++++++++++++++++++++-
 tools/kvm/kvm-cpu.c                              | 10 ++-
 tools/kvm/mmio.c                                 | 11 ++--
 tools/kvm/pci.c                                  |  3 +-
 tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h     |  2 +-
 tools/kvm/powerpc/kvm-cpu.c                      |  4 +-
 tools/kvm/powerpc/spapr_pci.h                    |  6 +-
 tools/kvm/virtio/9p.c                            |  3 +
 tools/kvm/virtio/blk.c                           | 31 +++++++--
 tools/kvm/virtio/console.c                       |  8 ++-
 tools/kvm/virtio/core.c                          | 59 +++++++++--------
 tools/kvm/virtio/mmio.c                          | 21 ++++--
 tools/kvm/virtio/net.c                           | 45 +++++++++++--
 tools/kvm/virtio/pci.c                           |  6 +-
 tools/kvm/x86/include/kvm/kvm-cpu-arch.h         |  4 +-
 24 files changed, 284 insertions(+), 75 deletions(-)

-- 
1.8.3.4

[PATCH v3 1/9] kvmtool: pass trapped vcpu to MMIO accessors

[Marc Zyngier]

In order to be able to find out about the endianness of a virtual
CPU, it is necessary to pass a pointer to the kvm_cpu structure
down to the MMIO accessors.

This patch just pushes such pointer as far as required for the
MMIO accessors to have a play with the vcpu.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/arm/include/arm-common/kvm-cpu-arch.h |  4 ++--
 tools/kvm/arm/kvm-cpu.c                         | 10 +++++-----
 tools/kvm/hw/pci-shmem.c                        |  2 +-
 tools/kvm/include/kvm/kvm.h                     |  4 ++--
 tools/kvm/kvm-cpu.c                             |  4 ++--
 tools/kvm/mmio.c                                | 11 ++++++-----
 tools/kvm/pci.c                                 |  3 ++-
 tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h    |  2 +-
 tools/kvm/powerpc/kvm-cpu.c                     |  4 ++--
 tools/kvm/powerpc/spapr_pci.h                   |  6 +++---
 tools/kvm/virtio/mmio.c                         | 18 +++++++++++-------
 tools/kvm/virtio/pci.c                          |  6 ++++--
 tools/kvm/x86/include/kvm/kvm-cpu-arch.h        |  4 ++--
 13 files changed, 43 insertions(+), 35 deletions(-)

diff --git a/tools/kvm/arm/include/arm-common/kvm-cpu-arch.h b/tools/kvm/arm/include/arm-common/kvm-cpu-arch.h
index bef1761..355a02d 100644
--- a/tools/kvm/arm/include/arm-common/kvm-cpu-arch.h
+++ b/tools/kvm/arm/include/arm-common/kvm-cpu-arch.h
@@ -42,8 +42,8 @@ static inline bool kvm_cpu__emulate_io(struct kvm *kvm, u16 port, void *data,
 	return false;
 }
 
-bool kvm_cpu__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len,
-			   u8 is_write);
+bool kvm_cpu__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data,
+			   u32 len, u8 is_write);
 
 unsigned long kvm_cpu__get_vcpu_mpidr(struct kvm_cpu *vcpu);
 
diff --git a/tools/kvm/arm/kvm-cpu.c b/tools/kvm/arm/kvm-cpu.c
index 9c9616f..53afa35 100644
--- a/tools/kvm/arm/kvm-cpu.c
+++ b/tools/kvm/arm/kvm-cpu.c
@@ -98,17 +98,17 @@ bool kvm_cpu__handle_exit(struct kvm_cpu *vcpu)
 	return false;
 }
 
-bool kvm_cpu__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len,
-			   u8 is_write)
+bool kvm_cpu__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data,
+			   u32 len, u8 is_write)
 {
 	if (arm_addr_in_virtio_mmio_region(phys_addr)) {
-		return kvm__emulate_mmio(kvm, phys_addr, data, len, is_write);
+		return kvm__emulate_mmio(vcpu, phys_addr, data, len, is_write);
 	} else if (arm_addr_in_ioport_region(phys_addr)) {
 		int direction = is_write ? KVM_EXIT_IO_OUT : KVM_EXIT_IO_IN;
 		u16 port = (phys_addr - KVM_IOPORT_AREA) & USHRT_MAX;
-		return kvm__emulate_io(kvm, port, data, direction, len, 1);
+		return kvm__emulate_io(vcpu->kvm, port, data, direction, len, 1);
 	} else if (arm_addr_in_pci_region(phys_addr)) {
-		return kvm__emulate_mmio(kvm, phys_addr, data, len, is_write);
+		return kvm__emulate_mmio(vcpu, phys_addr, data, len, is_write);
 	}
 
 	return false;
diff --git a/tools/kvm/hw/pci-shmem.c b/tools/kvm/hw/pci-shmem.c
index 34de747..4b837eb 100644
--- a/tools/kvm/hw/pci-shmem.c
+++ b/tools/kvm/hw/pci-shmem.c
@@ -105,7 +105,7 @@ static struct ioport_operations shmem_pci__io_ops = {
 	.io_out	= shmem_pci__io_out,
 };
 
-static void callback_mmio_msix(u64 addr, u8 *data, u32 len, u8 is_write, void *ptr)
+static void callback_mmio_msix(struct kvm_cpu *vcpu, u64 addr, u8 *data, u32 len, u8 is_write, void *ptr)
 {
 	void *mem;
 
diff --git a/tools/kvm/include/kvm/kvm.h b/tools/kvm/include/kvm/kvm.h
index d05b936..f1b71a0 100644
--- a/tools/kvm/include/kvm/kvm.h
+++ b/tools/kvm/include/kvm/kvm.h
@@ -84,10 +84,10 @@ int kvm_timer__exit(struct kvm *kvm);
 void kvm__irq_line(struct kvm *kvm, int irq, int level);
 void kvm__irq_trigger(struct kvm *kvm, int irq);
 bool kvm__emulate_io(struct kvm *kvm, u16 port, void *data, int direction, int size, u32 count);
-bool kvm__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8 is_write);
+bool kvm__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data, u32 len, u8 is_write);
 int kvm__register_mem(struct kvm *kvm, u64 guest_phys, u64 size, void *userspace_addr);
 int kvm__register_mmio(struct kvm *kvm, u64 phys_addr, u64 phys_addr_len, bool coalesce,
-			void (*mmio_fn)(u64 addr, u8 *data, u32 len, u8 is_write, void *ptr),
+		       void (*mmio_fn)(struct kvm_cpu *vcpu, u64 addr, u8 *data, u32 len, u8 is_write, void *ptr),
 			void *ptr);
 bool kvm__deregister_mmio(struct kvm *kvm, u64 phys_addr);
 void kvm__pause(struct kvm *kvm);
diff --git a/tools/kvm/kvm-cpu.c b/tools/kvm/kvm-cpu.c
index be05c49..5c70b00 100644
--- a/tools/kvm/kvm-cpu.c
+++ b/tools/kvm/kvm-cpu.c
@@ -54,7 +54,7 @@ static void kvm_cpu__handle_coalesced_mmio(struct kvm_cpu *cpu)
 		while (cpu->ring->first != cpu->ring->last) {
 			struct kvm_coalesced_mmio *m;
 			m = &cpu->ring->coalesced_mmio[cpu->ring->first];
-			kvm_cpu__emulate_mmio(cpu->kvm,
+			kvm_cpu__emulate_mmio(cpu,
 					      m->phys_addr,
 					      m->data,
 					      m->len,
@@ -138,7 +138,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
 			 */
 			kvm_cpu__handle_coalesced_mmio(cpu);
 
-			ret = kvm_cpu__emulate_mmio(cpu->kvm,
+			ret = kvm_cpu__emulate_mmio(cpu,
 						    cpu->kvm_run->mmio.phys_addr,
 						    cpu->kvm_run->mmio.data,
 						    cpu->kvm_run->mmio.len,
diff --git a/tools/kvm/mmio.c b/tools/kvm/mmio.c
index 5d65d28..705ce27f 100644
--- a/tools/kvm/mmio.c
+++ b/tools/kvm/mmio.c
@@ -1,4 +1,5 @@
 #include "kvm/kvm.h"
+#include "kvm/kvm-cpu.h"
 #include "kvm/rbtree-interval.h"
 #include "kvm/brlock.h"
 
@@ -16,7 +17,7 @@
 
 struct mmio_mapping {
 	struct rb_int_node	node;
-	void			(*mmio_fn)(u64 addr, u8 *data, u32 len, u8 is_write, void *ptr);
+	void			(*mmio_fn)(struct kvm_cpu *vcpu, u64 addr, u8 *data, u32 len, u8 is_write, void *ptr);
 	void			*ptr;
 };
 
@@ -59,7 +60,7 @@ static const char *to_direction(u8 is_write)
 }
 
 int kvm__register_mmio(struct kvm *kvm, u64 phys_addr, u64 phys_addr_len, bool coalesce,
-			void (*mmio_fn)(u64 addr, u8 *data, u32 len, u8 is_write, void *ptr),
+		       void (*mmio_fn)(struct kvm_cpu *vcpu, u64 addr, u8 *data, u32 len, u8 is_write, void *ptr),
 			void *ptr)
 {
 	struct mmio_mapping *mmio;
@@ -119,7 +120,7 @@ bool kvm__deregister_mmio(struct kvm *kvm, u64 phys_addr)
 	return true;
 }
 
-bool kvm__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8 is_write)
+bool kvm__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data, u32 len, u8 is_write)
 {
 	struct mmio_mapping *mmio;
 
@@ -127,9 +128,9 @@ bool kvm__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8 is_
 	mmio = mmio_search(&mmio_tree, phys_addr, len);
 
 	if (mmio)
-		mmio->mmio_fn(phys_addr, data, len, is_write, mmio->ptr);
+		mmio->mmio_fn(vcpu, phys_addr, data, len, is_write, mmio->ptr);
 	else {
-		if (kvm->cfg.mmio_debug)
+		if (vcpu->kvm->cfg.mmio_debug)
 			fprintf(stderr, "Warning: Ignoring MMIO %s at %016llx (length %u)\n",
 				to_direction(is_write), phys_addr, len);
 	}
diff --git a/tools/kvm/pci.c b/tools/kvm/pci.c
index c2da152..5d60585 100644
--- a/tools/kvm/pci.c
+++ b/tools/kvm/pci.c
@@ -179,7 +179,8 @@ void pci__config_rd(struct kvm *kvm, union pci_config_address addr, void *data,
 	}
 }
 
-static void pci_config_mmio_access(u64 addr, u8 *data, u32 len, u8 is_write, void *kvm)
+static void pci_config_mmio_access(struct kvm_cpu *vcpu, u64 addr, u8 *data,
+				   u32 len, u8 is_write, void *kvm)
 {
 	union pci_config_address cfg_addr;
 
diff --git a/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h b/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h
index 7520c04..82cb5987 100644
--- a/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h
+++ b/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h
@@ -71,6 +71,6 @@ static inline bool kvm_cpu__emulate_io(struct kvm *kvm, u16 port, void *data, in
 	return false;
 }
 
-bool kvm_cpu__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8 is_write);
+bool kvm_cpu__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data, u32 len, u8 is_write);
 
 #endif /* KVM__KVM_CPU_ARCH_H */
diff --git a/tools/kvm/powerpc/kvm-cpu.c b/tools/kvm/powerpc/kvm-cpu.c
index 8f825bd..2f5cfc6 100644
--- a/tools/kvm/powerpc/kvm-cpu.c
+++ b/tools/kvm/powerpc/kvm-cpu.c
@@ -193,7 +193,7 @@ bool kvm_cpu__handle_exit(struct kvm_cpu *vcpu)
 	return ret;
 }
 
-bool kvm_cpu__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8 is_write)
+bool kvm_cpu__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data, u32 len, u8 is_write)
 {
 	/*
 	 * FIXME: This function will need to be split in order to support
@@ -204,7 +204,7 @@ bool kvm_cpu__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8
 
 	if ((phys_addr >= SPAPR_PCI_WIN_START) &&
 	    (phys_addr < SPAPR_PCI_WIN_END)) {
-		ret = spapr_phb_mmio(kvm, phys_addr, data, len, is_write);
+		ret = spapr_phb_mmio(vcpu, phys_addr, data, len, is_write);
 	} else {
 		pr_warning("MMIO %s unknown address %llx (size %d)!\n",
 			   is_write ? "write to" : "read from",
diff --git a/tools/kvm/powerpc/spapr_pci.h b/tools/kvm/powerpc/spapr_pci.h
index 48b221c..f9cb42f 100644
--- a/tools/kvm/powerpc/spapr_pci.h
+++ b/tools/kvm/powerpc/spapr_pci.h
@@ -36,19 +36,19 @@ int spapr_populate_pci_devices(struct kvm *kvm,
                                uint32_t xics_phandle,
                                void *fdt);
 
-static inline bool spapr_phb_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8 is_write)
+static inline bool spapr_phb_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data, u32 len, u8 is_write)
 {
 	if ((phys_addr >= SPAPR_PCI_IO_WIN_ADDR) &&
 	    (phys_addr < SPAPR_PCI_IO_WIN_ADDR +
 	     SPAPR_PCI_IO_WIN_SIZE)) {
-		return kvm__emulate_io(kvm, phys_addr - SPAPR_PCI_IO_WIN_ADDR,
+		return kvm__emulate_io(vcpu->kvm, phys_addr - SPAPR_PCI_IO_WIN_ADDR,
 				       data, is_write ? KVM_EXIT_IO_OUT :
 				       KVM_EXIT_IO_IN,
 				       len, 1);
 	} else if ((phys_addr >= SPAPR_PCI_MEM_WIN_ADDR) &&
 		   (phys_addr < SPAPR_PCI_MEM_WIN_ADDR +
 		    SPAPR_PCI_MEM_WIN_SIZE)) {
-		return kvm__emulate_mmio(kvm, phys_addr - SPAPR_PCI_MEM_WIN_ADDR,
+		return kvm__emulate_mmio(vcpu, phys_addr - SPAPR_PCI_MEM_WIN_ADDR,
 					 data, len, is_write);
 	}
 	return false;
diff --git a/tools/kvm/virtio/mmio.c b/tools/kvm/virtio/mmio.c
index e1fca2b..9d385e2 100644
--- a/tools/kvm/virtio/mmio.c
+++ b/tools/kvm/virtio/mmio.c
@@ -88,7 +88,8 @@ int virtio_mmio_signal_config(struct kvm *kvm, struct virtio_device *vdev)
 	return 0;
 }
 
-static void virtio_mmio_device_specific(u64 addr, u8 *data, u32 len,
+static void virtio_mmio_device_specific(struct kvm_cpu *vcpu,
+					u64 addr, u8 *data, u32 len,
 					u8 is_write, struct virtio_device *vdev)
 {
 	struct virtio_mmio *vmmio = vdev->virtio;
@@ -104,7 +105,8 @@ static void virtio_mmio_device_specific(u64 addr, u8 *data, u32 len,
 	}
 }
 
-static void virtio_mmio_config_in(u64 addr, void *data, u32 len,
+static void virtio_mmio_config_in(struct kvm_cpu *vcpu,
+				  u64 addr, void *data, u32 len,
 				  struct virtio_device *vdev)
 {
 	struct virtio_mmio *vmmio = vdev->virtio;
@@ -140,7 +142,8 @@ static void virtio_mmio_config_in(u64 addr, void *data, u32 len,
 	}
 }
 
-static void virtio_mmio_config_out(u64 addr, void *data, u32 len,
+static void virtio_mmio_config_out(struct kvm_cpu *vcpu,
+				   u64 addr, void *data, u32 len,
 				   struct virtio_device *vdev)
 {
 	struct virtio_mmio *vmmio = vdev->virtio;
@@ -202,7 +205,8 @@ static void virtio_mmio_config_out(u64 addr, void *data, u32 len,
 	};
 }
 
-static void virtio_mmio_mmio_callback(u64 addr, u8 *data, u32 len,
+static void virtio_mmio_mmio_callback(struct kvm_cpu *vcpu,
+				      u64 addr, u8 *data, u32 len,
 				      u8 is_write, void *ptr)
 {
 	struct virtio_device *vdev = ptr;
@@ -211,14 +215,14 @@ static void virtio_mmio_mmio_callback(u64 addr, u8 *data, u32 len,
 
 	if (offset >= VIRTIO_MMIO_CONFIG) {
 		offset -= VIRTIO_MMIO_CONFIG;
-		virtio_mmio_device_specific(offset, data, len, is_write, ptr);
+		virtio_mmio_device_specific(vcpu, offset, data, len, is_write, ptr);
 		return;
 	}
 
 	if (is_write)
-		virtio_mmio_config_out(offset, data, len, ptr);
+		virtio_mmio_config_out(vcpu, offset, data, len, ptr);
 	else
-		virtio_mmio_config_in(offset, data, len, ptr);
+		virtio_mmio_config_in(vcpu, offset, data, len, ptr);
 }
 
 #ifdef CONFIG_HAS_LIBFDT
diff --git a/tools/kvm/virtio/pci.c b/tools/kvm/virtio/pci.c
index 665d492..57ccde6 100644
--- a/tools/kvm/virtio/pci.c
+++ b/tools/kvm/virtio/pci.c
@@ -240,7 +240,8 @@ static struct ioport_operations virtio_pci__io_ops = {
 	.io_out	= virtio_pci__io_out,
 };
 
-static void virtio_pci__msix_mmio_callback(u64 addr, u8 *data, u32 len,
+static void virtio_pci__msix_mmio_callback(struct kvm_cpu *vcpu,
+					   u64 addr, u8 *data, u32 len,
 					   u8 is_write, void *ptr)
 {
 	struct virtio_pci *vpci = ptr;
@@ -321,7 +322,8 @@ int virtio_pci__signal_config(struct kvm *kvm, struct virtio_device *vdev)
 	return 0;
 }
 
-static void virtio_pci__io_mmio_callback(u64 addr, u8 *data, u32 len,
+static void virtio_pci__io_mmio_callback(struct kvm_cpu *vcpu,
+					 u64 addr, u8 *data, u32 len,
 					 u8 is_write, void *ptr)
 {
 	struct virtio_pci *vpci = ptr;
diff --git a/tools/kvm/x86/include/kvm/kvm-cpu-arch.h b/tools/kvm/x86/include/kvm/kvm-cpu-arch.h
index 198efe6..fd86b4d 100644
--- a/tools/kvm/x86/include/kvm/kvm-cpu-arch.h
+++ b/tools/kvm/x86/include/kvm/kvm-cpu-arch.h
@@ -41,9 +41,9 @@ static inline bool kvm_cpu__emulate_io(struct kvm *kvm, u16 port, void *data, in
 	return kvm__emulate_io(kvm, port, data, direction, size, count);
 }
 
-static inline bool kvm_cpu__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len, u8 is_write)
+static inline bool kvm_cpu__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data, u32 len, u8 is_write)
 {
-	return kvm__emulate_mmio(kvm, phys_addr, data, len, is_write);
+	return kvm__emulate_mmio(vcpu, phys_addr, data, len, is_write);
 }
 
 #endif /* KVM__KVM_CPU_ARCH_H */
-- 
1.8.3.4

[PATCH v3 2/9] kvmtool: virt_queue configuration based on endianness

[Marc Zyngier]

Define a simple infrastructure to configure a virt_queue
depending on the guest endianness, as reported by the feature
flags. At this stage, the endianness is always the host's.

Wrap all accesses to virt_queue data structures shared between
host and guest with byte swapping helpers.

Should the architecture only support one endianness, these helpers
are reduced to the identity function.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/include/kvm/virtio.h | 74 ++++++++++++++++++++++++++++++++++++++++--
 tools/kvm/virtio/core.c        | 59 +++++++++++++++++++--------------
 2 files changed, 105 insertions(+), 28 deletions(-)

diff --git a/tools/kvm/include/kvm/virtio.h b/tools/kvm/include/kvm/virtio.h
index 820b94a..f6bddd9 100644
--- a/tools/kvm/include/kvm/virtio.h
+++ b/tools/kvm/include/kvm/virtio.h
@@ -1,6 +1,8 @@
 #ifndef KVM__VIRTIO_H
 #define KVM__VIRTIO_H
 
+#include <endian.h>
+
 #include <linux/virtio_ring.h>
 #include <linux/virtio_pci.h>
 
@@ -15,6 +17,10 @@
 #define VIRTIO_PCI_O_CONFIG	0
 #define VIRTIO_PCI_O_MSIX	1
 
+#define VIRTIO_ENDIAN_HOST	0
+#define VIRTIO_ENDIAN_LE	(1 << 0)
+#define VIRTIO_ENDIAN_BE	(1 << 1)
+
 struct virt_queue {
 	struct vring	vring;
 	u32		pfn;
@@ -24,9 +30,71 @@ struct virt_queue {
 	u16		last_used_signalled;
 };
 
+/*
+ * The default policy is not to cope with the guest endianness.
+ * It also helps not breaking archs that do not care about supporting
+ * such a configuration.
+ */
+#ifndef VIRTIO_RING_ENDIAN
+#define VIRTIO_RING_ENDIAN VIRTIO_ENDIAN_HOST
+#endif
+
+#if (VIRTIO_RING_ENDIAN & (VIRTIO_ENDIAN_LE | VIRTIO_ENDIAN_BE))
+
+static inline __u16 __virtio_g2h_u16(u16 endian, __u16 val)
+{
+	return (endian == VIRTIO_ENDIAN_LE) ? le16toh(val) : be16toh(val);
+}
+
+static inline __u16 __virtio_h2g_u16(u16 endian, __u16 val)
+{
+	return (endian == VIRTIO_ENDIAN_LE) ? htole16(val) : htobe16(val);
+}
+
+static inline __u32 __virtio_g2h_u32(u16 endian, __u32 val)
+{
+	return (endian == VIRTIO_ENDIAN_LE) ? le32toh(val) : be32toh(val);
+}
+
+static inline __u32 __virtio_h2g_u32(u16 endian, __u32 val)
+{
+	return (endian == VIRTIO_ENDIAN_LE) ? htole32(val) : htobe32(val);
+}
+
+static inline __u64 __virtio_g2h_u64(u16 endian, __u64 val)
+{
+	return (endian == VIRTIO_ENDIAN_LE) ? le64toh(val) : be64toh(val);
+}
+
+static inline __u64 __virtio_h2g_u64(u16 endian, __u64 val)
+{
+	return (endian == VIRTIO_ENDIAN_LE) ? htole64(val) : htobe64(val);
+}
+
+#define virtio_guest_to_host_u16(x, v)	__virtio_g2h_u16((x)->endian, (v))
+#define virtio_host_to_guest_u16(x, v)	__virtio_h2g_u16((x)->endian, (v))
+#define virtio_guest_to_host_u32(x, v)	__virtio_g2h_u32((x)->endian, (v))
+#define virtio_host_to_guest_u32(x, v)	__virtio_h2g_u32((x)->endian, (v))
+#define virtio_guest_to_host_u64(x, v)	__virtio_g2h_u64((x)->endian, (v))
+#define virtio_host_to_guest_u64(x, v)	__virtio_h2g_u64((x)->endian, (v))
+
+#else
+
+#define virtio_guest_to_host_u16(x, v)	(v)
+#define virtio_host_to_guest_u16(x, v)	(v)
+#define virtio_guest_to_host_u32(x, v)	(v)
+#define virtio_host_to_guest_u32(x, v)	(v)
+#define virtio_guest_to_host_u64(x, v)	(v)
+#define virtio_host_to_guest_u64(x, v)	(v)
+
+#endif
+
 static inline u16 virt_queue__pop(struct virt_queue *queue)
 {
-	return queue->vring.avail->ring[queue->last_avail_idx++ % queue->vring.num];
+	__u16 guest_idx;
+
+	guest_idx = queue->vring.avail->ring[queue->last_avail_idx++ % queue->vring.num];
+	return virtio_guest_to_host_u16(queue, guest_idx);
 }
 
 static inline struct vring_desc *virt_queue__get_desc(struct virt_queue *queue, u16 desc_ndx)
@@ -39,8 +107,8 @@ static inline bool virt_queue__available(struct virt_queue *vq)
 	if (!vq->vring.avail)
 		return 0;
 
-	vring_avail_event(&vq->vring) = vq->last_avail_idx;
-	return vq->vring.avail->idx !=  vq->last_avail_idx;
+	vring_avail_event(&vq->vring) = virtio_host_to_guest_u16(vq, vq->last_avail_idx);
+	return virtio_guest_to_host_u16(vq, vq->vring.avail->idx) != vq->last_avail_idx;
 }
 
 struct vring_used_elem *virt_queue__set_used_elem(struct virt_queue *queue, u32 head, u32 len);
diff --git a/tools/kvm/virtio/core.c b/tools/kvm/virtio/core.c
index 2dfb828..9ae7887 100644
--- a/tools/kvm/virtio/core.c
+++ b/tools/kvm/virtio/core.c
@@ -15,10 +15,11 @@
 struct vring_used_elem *virt_queue__set_used_elem(struct virt_queue *queue, u32 head, u32 len)
 {
 	struct vring_used_elem *used_elem;
+	u16 idx = virtio_guest_to_host_u16(queue, queue->vring.used->idx);
 
-	used_elem	= &queue->vring.used->ring[queue->vring.used->idx % queue->vring.num];
-	used_elem->id	= head;
-	used_elem->len	= len;
+	used_elem	= &queue->vring.used->ring[idx % queue->vring.num];
+	used_elem->id	= virtio_host_to_guest_u32(queue, head);
+	used_elem->len	= virtio_host_to_guest_u32(queue, len);
 
 	/*
 	 * Use wmb to assure that used elem was updated with head and len.
@@ -26,7 +27,8 @@ struct vring_used_elem *virt_queue__set_used_elem(struct virt_queue *queue, u32
 	 * to pass the used element to the guest.
 	 */
 	wmb();
-	queue->vring.used->idx++;
+	idx++;
+	queue->vring.used->idx = virtio_host_to_guest_u16(queue, idx);
 
 	/*
 	 * Use wmb to assure used idx has been increased before we signal the guest.
@@ -38,22 +40,28 @@ struct vring_used_elem *virt_queue__set_used_elem(struct virt_queue *queue, u32
 	return used_elem;
 }
 
+static inline bool virt_desc__test_flag(struct virt_queue *vq,
+					struct vring_desc *desc, u16 flag)
+{
+	return !!(virtio_guest_to_host_u16(vq, desc->flags) & flag);
+}
+
 /*
  * Each buffer in the virtqueues is actually a chain of descriptors.  This
  * function returns the next descriptor in the chain, or vq->vring.num if we're
  * at the end.
  */
-static unsigned next_desc(struct vring_desc *desc,
+static unsigned next_desc(struct virt_queue *vq, struct vring_desc *desc,
 			  unsigned int i, unsigned int max)
 {
 	unsigned int next;
 
 	/* If this descriptor says it doesn't chain, we're done. */
-	if (!(desc[i].flags & VRING_DESC_F_NEXT))
+	if (!virt_desc__test_flag(vq, &desc[i], VRING_DESC_F_NEXT))
 		return max;
 
 	/* Check they're not leading us off end of descriptors. */
-	next = desc[i].next;
+	next = virtio_guest_to_host_u16(vq, desc[i].next);
 	/* Make sure compiler knows to grab that: we don't want it changing! */
 	wmb();
 
@@ -71,22 +79,23 @@ u16 virt_queue__get_head_iov(struct virt_queue *vq, struct iovec iov[], u16 *out
 	max = vq->vring.num;
 	desc = vq->vring.desc;
 
-	if (desc[idx].flags & VRING_DESC_F_INDIRECT) {
-		max = desc[idx].len / sizeof(struct vring_desc);
-		desc = guest_flat_to_host(kvm, desc[idx].addr);
+	if (virt_desc__test_flag(vq, &desc[idx], VRING_DESC_F_INDIRECT)) {
+		max = virtio_guest_to_host_u32(vq, desc[idx].len) / sizeof(struct vring_desc);
+		desc = guest_flat_to_host(kvm, virtio_guest_to_host_u64(vq, desc[idx].addr));
 		idx = 0;
 	}
 
 	do {
 		/* Grab the first descriptor, and check it's OK. */
-		iov[*out + *in].iov_len = desc[idx].len;
-		iov[*out + *in].iov_base = guest_flat_to_host(kvm, desc[idx].addr);
+		iov[*out + *in].iov_len = virtio_guest_to_host_u32(vq, desc[idx].len);
+		iov[*out + *in].iov_base = guest_flat_to_host(kvm,
+							      virtio_guest_to_host_u64(vq, desc[idx].addr));
 		/* If this is an input descriptor, increment that count. */
-		if (desc[idx].flags & VRING_DESC_F_WRITE)
+		if (virt_desc__test_flag(vq, &desc[idx], VRING_DESC_F_WRITE))
 			(*in)++;
 		else
 			(*out)++;
-	} while ((idx = next_desc(desc, idx, max)) != max);
+	} while ((idx = next_desc(vq, desc, idx, max)) != max);
 
 	return head;
 }
@@ -111,20 +120,20 @@ u16 virt_queue__get_inout_iov(struct kvm *kvm, struct virt_queue *queue,
 	idx = head = virt_queue__pop(queue);
 	*out = *in = 0;
 	do {
+		u64 addr;
 		desc = virt_queue__get_desc(queue, idx);
-		if (desc->flags & VRING_DESC_F_WRITE) {
-			in_iov[*in].iov_base = guest_flat_to_host(kvm,
-								  desc->addr);
-			in_iov[*in].iov_len = desc->len;
+		addr = virtio_guest_to_host_u64(queue, desc->addr);
+		if (virt_desc__test_flag(queue, desc, VRING_DESC_F_WRITE)) {
+			in_iov[*in].iov_base = guest_flat_to_host(kvm, addr);
+			in_iov[*in].iov_len = virtio_guest_to_host_u32(queue, desc->len);
 			(*in)++;
 		} else {
-			out_iov[*out].iov_base = guest_flat_to_host(kvm,
-								    desc->addr);
-			out_iov[*out].iov_len = desc->len;
+			out_iov[*out].iov_base = guest_flat_to_host(kvm, addr);
+			out_iov[*out].iov_len = virtio_guest_to_host_u32(queue, desc->len);
 			(*out)++;
 		}
-		if (desc->flags & VRING_DESC_F_NEXT)
-			idx = desc->next;
+		if (virt_desc__test_flag(queue, desc, VRING_DESC_F_NEXT))
+			idx = virtio_guest_to_host_u16(queue, desc->next);
 		else
 			break;
 	} while (1);
@@ -151,8 +160,8 @@ bool virtio_queue__should_signal(struct virt_queue *vq)
 	u16 old_idx, new_idx, event_idx;
 
 	old_idx		= vq->last_used_signalled;
-	new_idx		= vq->vring.used->idx;
-	event_idx	= vring_used_event(&vq->vring);
+	new_idx		= virtio_guest_to_host_u16(vq, vq->vring.used->idx);
+	event_idx	= virtio_guest_to_host_u16(vq, vring_used_event(&vq->vring));
 
 	if (vring_need_event(event_idx, new_idx, old_idx)) {
 		vq->last_used_signalled = new_idx;
-- 
1.8.3.4

[PATCH v3 3/9] kvmtool: sample CPU endianness on virtio-mmio device reset

[Marc Zyngier]

Save the CPU endianness when the device is reset. It is widely
assumed that the guest won't change its endianness after, or at
least not without reseting the device first.

A default implementation of the endianness sampling just returns
the default "host endianness" value so that unsuspecting architectures
are not affected.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/include/kvm/kvm-cpu.h | 1 +
 tools/kvm/include/kvm/virtio.h  | 1 +
 tools/kvm/kvm-cpu.c             | 6 ++++++
 tools/kvm/virtio/mmio.c         | 3 +++
 4 files changed, 11 insertions(+)

diff --git a/tools/kvm/include/kvm/kvm-cpu.h b/tools/kvm/include/kvm/kvm-cpu.h
index 0ece28c..aa0cb54 100644
--- a/tools/kvm/include/kvm/kvm-cpu.h
+++ b/tools/kvm/include/kvm/kvm-cpu.h
@@ -15,6 +15,7 @@ void kvm_cpu__run(struct kvm_cpu *vcpu);
 void kvm_cpu__reboot(struct kvm *kvm);
 int kvm_cpu__start(struct kvm_cpu *cpu);
 bool kvm_cpu__handle_exit(struct kvm_cpu *vcpu);
+int kvm_cpu__get_endianness(struct kvm_cpu *vcpu);
 
 int kvm_cpu__get_debug_fd(void);
 void kvm_cpu__set_debug_fd(int fd);
diff --git a/tools/kvm/include/kvm/virtio.h b/tools/kvm/include/kvm/virtio.h
index f6bddd9..1180a3e 100644
--- a/tools/kvm/include/kvm/virtio.h
+++ b/tools/kvm/include/kvm/virtio.h
@@ -132,6 +132,7 @@ struct virtio_device {
 	bool			use_vhost;
 	void			*virtio;
 	struct virtio_ops	*ops;
+	u16			endian;
 };
 
 struct virtio_ops {
diff --git a/tools/kvm/kvm-cpu.c b/tools/kvm/kvm-cpu.c
index 5c70b00..9575b32 100644
--- a/tools/kvm/kvm-cpu.c
+++ b/tools/kvm/kvm-cpu.c
@@ -3,6 +3,7 @@
 #include "kvm/symbol.h"
 #include "kvm/util.h"
 #include "kvm/kvm.h"
+#include "kvm/virtio.h"
 
 #include <sys/ioctl.h>
 #include <sys/mman.h>
@@ -14,6 +15,11 @@
 
 extern __thread struct kvm_cpu *current_kvm_cpu;
 
+int __attribute__((weak)) kvm_cpu__get_endianness(struct kvm_cpu *vcpu)
+{
+	return VIRTIO_ENDIAN_HOST;
+}
+
 void kvm_cpu__enable_singlestep(struct kvm_cpu *vcpu)
 {
 	struct kvm_guest_debug debug = {
diff --git a/tools/kvm/virtio/mmio.c b/tools/kvm/virtio/mmio.c
index 9d385e2..3a2bd62 100644
--- a/tools/kvm/virtio/mmio.c
+++ b/tools/kvm/virtio/mmio.c
@@ -4,6 +4,7 @@
 #include "kvm/ioport.h"
 #include "kvm/virtio.h"
 #include "kvm/kvm.h"
+#include "kvm/kvm-cpu.h"
 #include "kvm/irq.h"
 #include "kvm/fdt.h"
 
@@ -159,6 +160,8 @@ static void virtio_mmio_config_out(struct kvm_cpu *vcpu,
 		break;
 	case VIRTIO_MMIO_STATUS:
 		vmmio->hdr.status = ioport__read32(data);
+		if (!vmmio->hdr.status) /* Sample endianness on reset */
+			vdev->endian = kvm_cpu__get_endianness(vcpu);
 		if (vdev->ops->notify_status)
 			vdev->ops->notify_status(kvm, vmmio->dev, vmmio->hdr.status);
 		break;
-- 
1.8.3.4

[PATCH v3 4/9] kvmtool: add queue endianness initializer

[Marc Zyngier]

Add a utility function that transfers the endianness sampled
at device reset time to a queue being set up.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/include/kvm/virtio.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tools/kvm/include/kvm/virtio.h b/tools/kvm/include/kvm/virtio.h
index 1180a3e..8a9eab5 100644
--- a/tools/kvm/include/kvm/virtio.h
+++ b/tools/kvm/include/kvm/virtio.h
@@ -28,6 +28,7 @@ struct virt_queue {
 	   It's where we assume the next request index is at.  */
 	u16		last_avail_idx;
 	u16		last_used_signalled;
+	u16		endian;
 };
 
 /*
@@ -165,4 +166,10 @@ static inline void *virtio_get_vq(struct kvm *kvm, u32 pfn, u32 page_size)
 	return guest_flat_to_host(kvm, (u64)pfn * page_size);
 }
 
+static inline void virtio_init_device_vq(struct virtio_device *vdev,
+					 struct virt_queue *vq)
+{
+	vq->endian = vdev->endian;
+}
+
 #endif /* KVM__VIRTIO_H */
-- 
1.8.3.4

[PATCH v3 5/9] kvmtool: convert console backend to support bi-endianness

[Marc Zyngier]

Configure the queues to follow the guest endianness, and make sure
the configuration space is doing the same.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/virtio/console.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/kvm/virtio/console.c b/tools/kvm/virtio/console.c
index 0474e2b..384eac1 100644
--- a/tools/kvm/virtio/console.c
+++ b/tools/kvm/virtio/console.c
@@ -131,7 +131,12 @@ static u32 get_host_features(struct kvm *kvm, void *dev)
 
 static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 {
-	/* Unused */
+	struct con_dev *cdev = dev;
+	struct virtio_console_config *conf = &cdev->config;
+
+	conf->cols = virtio_host_to_guest_u16(&cdev->vdev, conf->cols);
+	conf->rows = virtio_host_to_guest_u16(&cdev->vdev, conf->rows);
+	conf->max_nr_ports = virtio_host_to_guest_u32(&cdev->vdev, conf->max_nr_ports);
 }
 
 static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
@@ -149,6 +154,7 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
 	p		= virtio_get_vq(kvm, queue->pfn, page_size);
 
 	vring_init(&queue->vring, VIRTIO_CONSOLE_QUEUE_SIZE, p, align);
+	virtio_init_device_vq(&cdev.vdev, queue);
 
 	if (vq == VIRTIO_CONSOLE_TX_QUEUE) {
 		thread_pool__init_job(&cdev.jobs[vq], kvm, virtio_console_handle_callback, queue);
-- 
1.8.3.4

[PATCH v3 6/9] kvmtool: convert 9p backend to support bi-endianness

[Marc Zyngier]

Configure the queues to follow the guest endianness, and make sure
the configuration space is doing the same.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/virtio/9p.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/kvm/virtio/9p.c b/tools/kvm/virtio/9p.c
index 847eddb..9073a1e 100644
--- a/tools/kvm/virtio/9p.c
+++ b/tools/kvm/virtio/9p.c
@@ -1252,8 +1252,10 @@ static u32 get_host_features(struct kvm *kvm, void *dev)
 static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 {
 	struct p9_dev *p9dev = dev;
+	struct virtio_9p_config *conf = p9dev->config;
 
 	p9dev->features = features;
+	conf->tag_len = virtio_host_to_guest_u16(&p9dev->vdev, conf->tag_len);
 }
 
 static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
@@ -1272,6 +1274,7 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
 	job		= &p9dev->jobs[vq];
 
 	vring_init(&queue->vring, VIRTQUEUE_NUM, p, align);
+	virtio_init_device_vq(&p9dev->vdev, queue);
 
 	*job		= (struct p9_dev_job) {
 		.vq		= queue,
-- 
1.8.3.4

[PATCH v3 7/9] kvmtool: convert blk backend to support bi-endianness

[Marc Zyngier]

Configure the queues to follow the guest endianness, and make sure
the configuration space is doing the same.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/virtio/blk.c | 31 +++++++++++++++++++++++++------
 1 file changed, 25 insertions(+), 6 deletions(-)

diff --git a/tools/kvm/virtio/blk.c b/tools/kvm/virtio/blk.c
index 4bed3a9..edfa8e6 100644
--- a/tools/kvm/virtio/blk.c
+++ b/tools/kvm/virtio/blk.c
@@ -77,13 +77,15 @@ void virtio_blk_complete(void *param, long len)
 		bdev->vdev.ops->signal_vq(req->kvm, &bdev->vdev, queueid);
 }
 
-static void virtio_blk_do_io_request(struct kvm *kvm, struct blk_dev_req *req)
+static void virtio_blk_do_io_request(struct kvm *kvm, struct virt_queue *vq, struct blk_dev_req *req)
 {
 	struct virtio_blk_outhdr *req_hdr;
 	ssize_t block_cnt;
 	struct blk_dev *bdev;
 	struct iovec *iov;
 	u16 out, in;
+	u32 type;
+	u64 sector;
 
 	block_cnt	= -1;
 	bdev		= req->bdev;
@@ -92,13 +94,16 @@ static void virtio_blk_do_io_request(struct kvm *kvm, struct blk_dev_req *req)
 	in		= req->in;
 	req_hdr		= iov[0].iov_base;
 
-	switch (req_hdr->type) {
+	type = virtio_guest_to_host_u32(vq, req_hdr->type);
+	sector = virtio_guest_to_host_u64(vq, req_hdr->sector);
+
+	switch (type) {
 	case VIRTIO_BLK_T_IN:
-		block_cnt = disk_image__read(bdev->disk, req_hdr->sector,
+		block_cnt = disk_image__read(bdev->disk, sector,
 				iov + 1, in + out - 2, req);
 		break;
 	case VIRTIO_BLK_T_OUT:
-		block_cnt = disk_image__write(bdev->disk, req_hdr->sector,
+		block_cnt = disk_image__write(bdev->disk, sector,
 				iov + 1, in + out - 2, req);
 		break;
 	case VIRTIO_BLK_T_FLUSH:
@@ -112,7 +117,7 @@ static void virtio_blk_do_io_request(struct kvm *kvm, struct blk_dev_req *req)
 		virtio_blk_complete(req, block_cnt);
 		break;
 	default:
-		pr_warning("request type %d", req_hdr->type);
+		pr_warning("request type %d", type);
 		block_cnt	= -1;
 		break;
 	}
@@ -130,7 +135,7 @@ static void virtio_blk_do_io(struct kvm *kvm, struct virt_queue *vq, struct blk_
 					&req->in, head, kvm);
 		req->vq		= vq;
 
-		virtio_blk_do_io_request(kvm, req);
+		virtio_blk_do_io_request(kvm, vq, req);
 	}
 }
 
@@ -152,8 +157,21 @@ static u32 get_host_features(struct kvm *kvm, void *dev)
 static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 {
 	struct blk_dev *bdev = dev;
+	struct virtio_blk_config *conf = &bdev->blk_config;
+	struct virtio_blk_geometry *geo = &conf->geometry;
 
 	bdev->features = features;
+
+	conf->capacity = virtio_host_to_guest_u64(&bdev->vdev, conf->capacity);
+	conf->size_max = virtio_host_to_guest_u32(&bdev->vdev, conf->size_max);
+	conf->seg_max = virtio_host_to_guest_u32(&bdev->vdev, conf->seg_max);
+
+	/* Geometry */
+	geo->cylinders = virtio_host_to_guest_u16(&bdev->vdev, geo->cylinders);
+
+	conf->blk_size = virtio_host_to_guest_u32(&bdev->vdev, conf->blk_size);
+	conf->min_io_size = virtio_host_to_guest_u16(&bdev->vdev, conf->min_io_size);
+	conf->opt_io_size = virtio_host_to_guest_u32(&bdev->vdev, conf->opt_io_size);
 }
 
 static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
@@ -170,6 +188,7 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
 	p		= virtio_get_vq(kvm, queue->pfn, page_size);
 
 	vring_init(&queue->vring, VIRTIO_BLK_QUEUE_SIZE, p, align);
+	virtio_init_device_vq(&bdev->vdev, queue);
 
 	return 0;
 }
-- 
1.8.3.4

[PATCH v3 8/9] kvmtool: convert net backend to support bi-endianness

[Marc Zyngier]

Configure the queues to follow the guest endianness, and make sure
the configuration space is doing the same.

Extra care is taken for the handling of the virtio_net_hdr structures
on both the TX and RX ends.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/virtio/net.c | 45 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 40 insertions(+), 5 deletions(-)

diff --git a/tools/kvm/virtio/net.c b/tools/kvm/virtio/net.c
index dbb4431..363ec73 100644
--- a/tools/kvm/virtio/net.c
+++ b/tools/kvm/virtio/net.c
@@ -73,6 +73,24 @@ static bool has_virtio_feature(struct net_dev *ndev, u32 feature)
 	return ndev->features & (1 << feature);
 }
 
+static void virtio_net_fix_tx_hdr(struct virtio_net_hdr *hdr, struct net_dev *ndev)
+{
+	hdr->hdr_len		= virtio_guest_to_host_u16(&ndev->vdev, hdr->hdr_len);
+	hdr->gso_size		= virtio_guest_to_host_u16(&ndev->vdev, hdr->gso_size);
+	hdr->csum_start		= virtio_guest_to_host_u16(&ndev->vdev, hdr->csum_start);
+	hdr->csum_offset	= virtio_guest_to_host_u16(&ndev->vdev, hdr->csum_offset);
+}
+
+static void virtio_net_fix_rx_hdr(struct virtio_net_hdr_mrg_rxbuf *hdr, struct net_dev *ndev)
+{
+	hdr->hdr.hdr_len	= virtio_host_to_guest_u16(&ndev->vdev, hdr->hdr.hdr_len);
+	hdr->hdr.gso_size	= virtio_host_to_guest_u16(&ndev->vdev, hdr->hdr.gso_size);
+	hdr->hdr.csum_start	= virtio_host_to_guest_u16(&ndev->vdev, hdr->hdr.csum_start);
+	hdr->hdr.csum_offset	= virtio_host_to_guest_u16(&ndev->vdev, hdr->hdr.csum_offset);
+	if (has_virtio_feature(ndev, VIRTIO_NET_F_MRG_RXBUF))
+		hdr->num_buffers	= virtio_host_to_guest_u16(&ndev->vdev, hdr->num_buffers);
+}
+
 static void *virtio_net_rx_thread(void *p)
 {
 	struct iovec iov[VIRTIO_NET_QUEUE_SIZE];
@@ -106,6 +124,7 @@ static void *virtio_net_rx_thread(void *p)
 				.iov_len  = sizeof(buffer),
 			};
 			struct virtio_net_hdr_mrg_rxbuf *hdr;
+			int i;
 
 			len = ndev->ops->rx(&dummy_iov, 1, ndev);
 			if (len < 0) {
@@ -114,16 +133,20 @@ static void *virtio_net_rx_thread(void *p)
 				goto out_err;
 			}
 
-			copied = 0;
+			copied = i = 0;
 			head = virt_queue__get_iov(vq, iov, &out, &in, kvm);
-			hdr = (void *)iov[0].iov_base;
+			hdr = iov[0].iov_base;
 			while (copied < len) {
 				size_t iovsize = min_t(size_t, len - copied, iov_size(iov, in));
 
 				memcpy_toiovec(iov, buffer + copied, iovsize);
 				copied += iovsize;
-				if (has_virtio_feature(ndev, VIRTIO_NET_F_MRG_RXBUF))
-					hdr->num_buffers++;
+				if (i++ == 0)
+					virtio_net_fix_rx_hdr(hdr, ndev);
+				if (has_virtio_feature(ndev, VIRTIO_NET_F_MRG_RXBUF)) {
+					u16 num_buffers = virtio_guest_to_host_u16(vq, hdr->num_buffers);
+					hdr->num_buffers = virtio_host_to_guest_u16(vq, num_buffers + 1);
+				}
 				virt_queue__set_used_elem(vq, head, iovsize);
 				if (copied == len)
 					break;
@@ -170,11 +193,14 @@ static void *virtio_net_tx_thread(void *p)
 		mutex_unlock(&ndev->io_lock[id]);
 
 		while (virt_queue__available(vq)) {
+			struct virtio_net_hdr *hdr;
 			head = virt_queue__get_iov(vq, iov, &out, &in, kvm);
+			hdr = iov[0].iov_base;
+			virtio_net_fix_tx_hdr(hdr, ndev);
 			len = ndev->ops->tx(iov, out, ndev);
 			if (len < 0) {
 				pr_warning("%s: tx on vq %u failed (%d)\n",
-						__func__, id, len);
+						__func__, id, errno);
 				goto out_err;
 			}
 
@@ -415,9 +441,14 @@ static int virtio_net__vhost_set_features(struct net_dev *ndev)
 static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 {
 	struct net_dev *ndev = dev;
+	struct virtio_net_config *conf = &ndev->config;
 
 	ndev->features = features;
 
+	conf->status = virtio_host_to_guest_u16(&ndev->vdev, conf->status);
+	conf->max_virtqueue_pairs = virtio_host_to_guest_u16(&ndev->vdev,
+							     conf->max_virtqueue_pairs);
+
 	if (ndev->mode == NET_MODE_TAP) {
 		if (!virtio_net__tap_init(ndev))
 			die_perror("You have requested a TAP device, but creation of one has failed because");
@@ -454,6 +485,7 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
 	p		= virtio_get_vq(kvm, queue->pfn, page_size);
 
 	vring_init(&queue->vring, VIRTIO_NET_QUEUE_SIZE, p, align);
+	virtio_init_device_vq(&ndev->vdev, queue);
 
 	mutex_init(&ndev->io_lock[vq]);
 	pthread_cond_init(&ndev->io_cond[vq], NULL);
@@ -470,6 +502,9 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
 		return 0;
 	}
 
+	if (queue->endian != VIRTIO_ENDIAN_HOST)
+		die_perror("VHOST requires VIRTIO_ENDIAN_HOST");
+
 	state.num = queue->vring.num;
 	r = ioctl(ndev->vhost_fd, VHOST_SET_VRING_NUM, &state);
 	if (r < 0)
-- 
1.8.3.4

[PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

Implement the kcm_cpu__get_endianness call for both AArch32 and
AArch64, and advertise the bi-endianness support.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 tools/kvm/arm/aarch32/kvm-cpu.c                  | 14 +++++++++++++
 tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h |  2 ++
 tools/kvm/arm/aarch64/kvm-cpu.c                  | 25 ++++++++++++++++++++++++
 tools/kvm/arm/include/arm-common/kvm-arch.h      |  2 ++
 4 files changed, 43 insertions(+)

diff --git a/tools/kvm/arm/aarch32/kvm-cpu.c b/tools/kvm/arm/aarch32/kvm-cpu.c
index bd71037..464b473 100644
--- a/tools/kvm/arm/aarch32/kvm-cpu.c
+++ b/tools/kvm/arm/aarch32/kvm-cpu.c
@@ -1,5 +1,6 @@
 #include "kvm/kvm-cpu.h"
 #include "kvm/kvm.h"
+#include "kvm/virtio.h"
 
 #include <asm/ptrace.h>
 
@@ -76,6 +77,19 @@ void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
 		die_perror("KVM_SET_ONE_REG failed (pc)");
 }
 
+int kvm_cpu__get_endianness(struct kvm_cpu *vcpu)
+{
+	struct kvm_one_reg reg;
+	u32 data;
+
+	reg.id = ARM_CORE_REG(usr_regs.ARM_cpsr);
+	reg.addr = (u64)(unsigned long)&data;
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (cpsr)");
+
+	return (data & PSR_E_BIT) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
+}
+
 void kvm_cpu__show_code(struct kvm_cpu *vcpu)
 {
 	struct kvm_one_reg reg;
diff --git a/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
index 7d70c3b..ed7da45 100644
--- a/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
+++ b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
@@ -13,5 +13,7 @@
 #define ARM_MPIDR_HWID_BITMASK	0xFF00FFFFFFUL
 #define ARM_CPU_ID		3, 0, 0, 0
 #define ARM_CPU_ID_MPIDR	5
+#define ARM_CPU_CTRL		3, 0, 1, 0
+#define ARM_CPU_CTRL_SCTLR	0
 
 #endif /* KVM__KVM_CPU_ARCH_H */
diff --git a/tools/kvm/arm/aarch64/kvm-cpu.c b/tools/kvm/arm/aarch64/kvm-cpu.c
index 059e42c..b3ce2c8 100644
--- a/tools/kvm/arm/aarch64/kvm-cpu.c
+++ b/tools/kvm/arm/aarch64/kvm-cpu.c
@@ -1,12 +1,16 @@
 #include "kvm/kvm-cpu.h"
 #include "kvm/kvm.h"
+#include "kvm/virtio.h"
 
 #include <asm/ptrace.h>
 
 #define COMPAT_PSR_F_BIT	0x00000040
 #define COMPAT_PSR_I_BIT	0x00000080
+#define COMPAT_PSR_E_BIT	0x00000200
 #define COMPAT_PSR_MODE_SVC	0x00000013
 
+#define SCTLR_EL1_EE_MASK	(1 << 25)
+
 #define ARM64_CORE_REG(x)	(KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
 				 KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
@@ -133,6 +137,27 @@ void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
 		return reset_vcpu_aarch64(vcpu);
 }
 
+int kvm_cpu__get_endianness(struct kvm_cpu *vcpu)
+{
+	struct kvm_one_reg reg;
+	u64 data;
+
+	reg.id = ARM64_CORE_REG(regs.pstate);
+	reg.addr = (u64)&data;
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (spsr[EL1])");
+
+	if (data & PSR_MODE32_BIT)
+		return (data & COMPAT_PSR_E_BIT) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
+
+	reg.id = ARM64_SYS_REG(ARM_CPU_CTRL, ARM_CPU_CTRL_SCTLR); /* SCTLR_EL1 */
+	reg.addr = (u64)&data;
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
+
+	return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
+}
+
 void kvm_cpu__show_code(struct kvm_cpu *vcpu)
 {
 	struct kvm_one_reg reg;
diff --git a/tools/kvm/arm/include/arm-common/kvm-arch.h b/tools/kvm/arm/include/arm-common/kvm-arch.h
index b6c4bf8..5d2fab2 100644
--- a/tools/kvm/arm/include/arm-common/kvm-arch.h
+++ b/tools/kvm/arm/include/arm-common/kvm-arch.h
@@ -35,6 +35,8 @@
 #define VIRTIO_DEFAULT_TRANS(kvm)	\
 	((kvm)->cfg.arch.virtio_trans_pci ? VIRTIO_PCI : VIRTIO_MMIO)
 
+#define VIRTIO_RING_ENDIAN	(VIRTIO_ENDIAN_LE | VIRTIO_ENDIAN_BE)
+
 static inline bool arm_addr_in_ioport_region(u64 phys_addr)
 {
 	u64 limit = KVM_IOPORT_AREA + ARM_IOPORT_SIZE;
-- 
1.8.3.4

Re: [PATCH v3 0/9] kvmtool: handle guests of a different endianness

[Pekka Enberg]

On 4/24/14 9:17 PM, Marc Zyngier wrote:
> This patch series adds some infrastructure to kvmtool to allow a BE
> guest to use virtio-mmio on a LE host, provided that the architecture
> actually supports such madness.
>
> Not all the backend have been converted, only those I actually cared
> about. Converting them is pretty easy though, and will be done if the
> method is deemed acceptable.
>
> This has been tested on both arm and arm64 (I use this on a daily
> basis to test BE code). The corresponding kernel changes have all been
> merged.

Will, comments? I'm OK with this.

- Pekka

Re: [PATCH v3 0/9] kvmtool: handle guests of a different endianness

[Will Deacon]

Hi Pekka,

On Sat, May 03, 2014 at 08:07:59AM +0100, Pekka Enberg wrote:
> On 4/24/14 9:17 PM, Marc Zyngier wrote:
> > This patch series adds some infrastructure to kvmtool to allow a BE
> > guest to use virtio-mmio on a LE host, provided that the architecture
> > actually supports such madness.
> >
> > Not all the backend have been converted, only those I actually cared
> > about. Converting them is pretty easy though, and will be done if the
> > method is deemed acceptable.
> >
> > This has been tested on both arm and arm64 (I use this on a daily
> > basis to test BE code). The corresponding kernel changes have all been
> > merged.
> 
> Will, comments? I'm OK with this.

I'd like to take a closer look at the last patch, so please hold off merging
these for the moment.

Cheers,

Will

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Will Deacon]

Hi Marc,

On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
> Implement the kcm_cpu__get_endianness call for both AArch32 and

s/kcm/kvm/

> AArch64, and advertise the bi-endianness support.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  tools/kvm/arm/aarch32/kvm-cpu.c                  | 14 +++++++++++++
>  tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h |  2 ++
>  tools/kvm/arm/aarch64/kvm-cpu.c                  | 25 ++++++++++++++++++++++++
>  tools/kvm/arm/include/arm-common/kvm-arch.h      |  2 ++
>  4 files changed, 43 insertions(+)
> 
> diff --git a/tools/kvm/arm/aarch32/kvm-cpu.c b/tools/kvm/arm/aarch32/kvm-cpu.c
> index bd71037..464b473 100644
> --- a/tools/kvm/arm/aarch32/kvm-cpu.c
> +++ b/tools/kvm/arm/aarch32/kvm-cpu.c
> @@ -1,5 +1,6 @@
>  #include "kvm/kvm-cpu.h"
>  #include "kvm/kvm.h"
> +#include "kvm/virtio.h"
>  
>  #include <asm/ptrace.h>
>  
> @@ -76,6 +77,19 @@ void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
>  		die_perror("KVM_SET_ONE_REG failed (pc)");
>  }
>  
> +int kvm_cpu__get_endianness(struct kvm_cpu *vcpu)
> +{
> +	struct kvm_one_reg reg;
> +	u32 data;
> +
> +	reg.id = ARM_CORE_REG(usr_regs.ARM_cpsr);
> +	reg.addr = (u64)(unsigned long)&data;
> +	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
> +		die("KVM_GET_ONE_REG failed (cpsr)");
> +
> +	return (data & PSR_E_BIT) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
> +}
> +
>  void kvm_cpu__show_code(struct kvm_cpu *vcpu)
>  {
>  	struct kvm_one_reg reg;
> diff --git a/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
> index 7d70c3b..ed7da45 100644
> --- a/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
> +++ b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
> @@ -13,5 +13,7 @@
>  #define ARM_MPIDR_HWID_BITMASK	0xFF00FFFFFFUL
>  #define ARM_CPU_ID		3, 0, 0, 0
>  #define ARM_CPU_ID_MPIDR	5
> +#define ARM_CPU_CTRL		3, 0, 1, 0
> +#define ARM_CPU_CTRL_SCTLR	0
>  
>  #endif /* KVM__KVM_CPU_ARCH_H */
> diff --git a/tools/kvm/arm/aarch64/kvm-cpu.c b/tools/kvm/arm/aarch64/kvm-cpu.c
> index 059e42c..b3ce2c8 100644
> --- a/tools/kvm/arm/aarch64/kvm-cpu.c
> +++ b/tools/kvm/arm/aarch64/kvm-cpu.c
> @@ -1,12 +1,16 @@
>  #include "kvm/kvm-cpu.h"
>  #include "kvm/kvm.h"
> +#include "kvm/virtio.h"
>  
>  #include <asm/ptrace.h>
>  
>  #define COMPAT_PSR_F_BIT	0x00000040
>  #define COMPAT_PSR_I_BIT	0x00000080
> +#define COMPAT_PSR_E_BIT	0x00000200
>  #define COMPAT_PSR_MODE_SVC	0x00000013
>  
> +#define SCTLR_EL1_EE_MASK	(1 << 25)
> +
>  #define ARM64_CORE_REG(x)	(KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
>  				 KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>  
> @@ -133,6 +137,27 @@ void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
>  		return reset_vcpu_aarch64(vcpu);
>  }
>  
> +int kvm_cpu__get_endianness(struct kvm_cpu *vcpu)
> +{
> +	struct kvm_one_reg reg;
> +	u64 data;
> +
> +	reg.id = ARM64_CORE_REG(regs.pstate);
> +	reg.addr = (u64)&data;
> +	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
> +		die("KVM_GET_ONE_REG failed (spsr[EL1])");

This bit hurt for a while ;) Can you add a comment mentioning SETEND for
AArch32 guests please?

> +	if (data & PSR_MODE32_BIT)
> +		return (data & COMPAT_PSR_E_BIT) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
> +
> +	reg.id = ARM64_SYS_REG(ARM_CPU_CTRL, ARM_CPU_CTRL_SCTLR); /* SCTLR_EL1 */

We should probably just rename ARM_CPU_CTRL_SCTLR that to include the _EL1
suffix.

> +	reg.addr = (u64)&data;
> +	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
> +		die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
> +
> +	return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;

This rules out guests where userspace and kernelspace can run with different
endinness. Whilst Linux doesn't currently do this, can we support it here?
It all gets a bit hairy if the guest is using a stage-1 SMMU to let
userspace play with a virtio device...

Will

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

Hi Will,

On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
> Hi Marc,
>
> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>> Implement the kcm_cpu__get_endianness call for both AArch32 and
>
> s/kcm/kvm/

Are you saying that I hace fat fingers? ;-)

>> AArch64, and advertise the bi-endianness support.
>> 
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  tools/kvm/arm/aarch32/kvm-cpu.c                  | 14 +++++++++++++
>>  tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h |  2 ++
>>  tools/kvm/arm/aarch64/kvm-cpu.c                  | 25 ++++++++++++++++++++++++
>>  tools/kvm/arm/include/arm-common/kvm-arch.h      |  2 ++
>>  4 files changed, 43 insertions(+)
>> 
>> diff --git a/tools/kvm/arm/aarch32/kvm-cpu.c b/tools/kvm/arm/aarch32/kvm-cpu.c
>> index bd71037..464b473 100644
>> --- a/tools/kvm/arm/aarch32/kvm-cpu.c
>> +++ b/tools/kvm/arm/aarch32/kvm-cpu.c
>> @@ -1,5 +1,6 @@
>>  #include "kvm/kvm-cpu.h"
>>  #include "kvm/kvm.h"
>> +#include "kvm/virtio.h"
>>  
>>  #include <asm/ptrace.h>
>>  
>> @@ -76,6 +77,19 @@ void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
>>  		die_perror("KVM_SET_ONE_REG failed (pc)");
>>  }
>>  
>> +int kvm_cpu__get_endianness(struct kvm_cpu *vcpu)
>> +{
>> +	struct kvm_one_reg reg;
>> +	u32 data;
>> +
>> +	reg.id = ARM_CORE_REG(usr_regs.ARM_cpsr);
>> +	reg.addr = (u64)(unsigned long)&data;
>> +	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>> +		die("KVM_GET_ONE_REG failed (cpsr)");
>> +
>> +	return (data & PSR_E_BIT) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>> +}
>> +
>>  void kvm_cpu__show_code(struct kvm_cpu *vcpu)
>>  {
>>  	struct kvm_one_reg reg;
>> diff --git a/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
>> index 7d70c3b..ed7da45 100644
>> --- a/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
>> +++ b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
>> @@ -13,5 +13,7 @@
>>  #define ARM_MPIDR_HWID_BITMASK	0xFF00FFFFFFUL
>>  #define ARM_CPU_ID		3, 0, 0, 0
>>  #define ARM_CPU_ID_MPIDR	5
>> +#define ARM_CPU_CTRL		3, 0, 1, 0
>> +#define ARM_CPU_CTRL_SCTLR	0
>>  
>>  #endif /* KVM__KVM_CPU_ARCH_H */
>> diff --git a/tools/kvm/arm/aarch64/kvm-cpu.c b/tools/kvm/arm/aarch64/kvm-cpu.c
>> index 059e42c..b3ce2c8 100644
>> --- a/tools/kvm/arm/aarch64/kvm-cpu.c
>> +++ b/tools/kvm/arm/aarch64/kvm-cpu.c
>> @@ -1,12 +1,16 @@
>>  #include "kvm/kvm-cpu.h"
>>  #include "kvm/kvm.h"
>> +#include "kvm/virtio.h"
>>  
>>  #include <asm/ptrace.h>
>>  
>>  #define COMPAT_PSR_F_BIT	0x00000040
>>  #define COMPAT_PSR_I_BIT	0x00000080
>> +#define COMPAT_PSR_E_BIT	0x00000200
>>  #define COMPAT_PSR_MODE_SVC	0x00000013
>>  
>> +#define SCTLR_EL1_EE_MASK	(1 << 25)
>> +
>>  #define ARM64_CORE_REG(x)	(KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
>>  				 KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>>  
>> @@ -133,6 +137,27 @@ void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
>>  		return reset_vcpu_aarch64(vcpu);
>>  }
>>  
>> +int kvm_cpu__get_endianness(struct kvm_cpu *vcpu)
>> +{
>> +	struct kvm_one_reg reg;
>> +	u64 data;
>> +
>> +	reg.id = ARM64_CORE_REG(regs.pstate);
>> +	reg.addr = (u64)&data;
>> +	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>> +		die("KVM_GET_ONE_REG failed (spsr[EL1])");
>
> This bit hurt for a while ;) Can you add a comment mentioning SETEND for
> AArch32 guests please?

Sure.

>> +	if (data & PSR_MODE32_BIT)
>> +		return (data & COMPAT_PSR_E_BIT) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>> +
>> +	reg.id = ARM64_SYS_REG(ARM_CPU_CTRL, ARM_CPU_CTRL_SCTLR); /* SCTLR_EL1 */
>
> We should probably just rename ARM_CPU_CTRL_SCTLR that to include the _EL1
> suffix.

Fair enough.

>> +	reg.addr = (u64)&data;
>> +	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>> +		die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>> +
>> +	return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>
> This rules out guests where userspace and kernelspace can run with different
> endinness. Whilst Linux doesn't currently do this, can we support it here?
> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
> userspace play with a virtio device...

Yeah, I suppose we could check either EE or E0 depending on the mode
when the access was made. We already have all the information, just need
to handle the case. I'll respin the series.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Peter Maydell]

On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>> +    reg.addr = (u64)&data;
>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>> +
>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>>
>> This rules out guests where userspace and kernelspace can run with different
>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>> userspace play with a virtio device...
>
> Yeah, I suppose we could check either EE or E0 depending on the mode
> when the access was made. We already have all the information, just need
> to handle the case. I'll respin the series.

Hi Marc :-)

How virtio implementations should determine their endianness is
a spec question, I think; at any rate QEMU and kvmtool ought to
agree on how it's done. I think the most recent suggestion on the
QEMU mailing list (for PPC) is that we should care about the
guest kernel endianness, but I don't know if anybody thought of
the pass-through-to-userspace usecase...

thanks
-- PMM

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Peter Maydell]

On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>>> +    reg.addr = (u64)&data;
>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>>> +
>>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>>>
>>> This rules out guests where userspace and kernelspace can run with different
>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>>> userspace play with a virtio device...
>>
>> Yeah, I suppose we could check either EE or E0 depending on the mode
>> when the access was made. We already have all the information, just need
>> to handle the case. I'll respin the series.

> How virtio implementations should determine their endianness is
> a spec question, I think; at any rate QEMU and kvmtool ought to
> agree on how it's done. I think the most recent suggestion on the
> QEMU mailing list (for PPC) is that we should care about the
> guest kernel endianness, but I don't know if anybody thought of
> the pass-through-to-userspace usecase...

Current opinion on the qemu-devel thread seems to be that we
should just define that the endianness of the virtio device is
the endianness of the guest kernel at the point where the guest
triggers a reset of the virtio device by writing zero the QueuePFN
or Status registers.

thanks
-- PMM

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Alexander Graf]

> Am 07.05.2014 um 11:34 schrieb Peter Maydell <peter.maydell@linaro.org>:
> 
>> On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
>>>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>>>> +    reg.addr = (u64)&data;
>>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>>>> +
>>>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>>>> 
>>>> This rules out guests where userspace and kernelspace can run with different
>>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>>>> userspace play with a virtio device...
>>> 
>>> Yeah, I suppose we could check either EE or E0 depending on the mode
>>> when the access was made. We already have all the information, just need
>>> to handle the case. I'll respin the series.
> 
>> How virtio implementations should determine their endianness is
>> a spec question, I think; at any rate QEMU and kvmtool ought to
>> agree on how it's done. I think the most recent suggestion on the
>> QEMU mailing list (for PPC) is that we should care about the
>> guest kernel endianness, but I don't know if anybody thought of
>> the pass-through-to-userspace usecase...
> 
> Current opinion on the qemu-devel thread seems to be that we
> should just define that the endianness of the virtio device is
> the endianness of the guest kernel at the point where the guest
> triggers a reset of the virtio device by writing zero the QueuePFN
> or Status registers.

Virtio by design has full access to guest physical memory. It doesn't route DMA via PCI. So user space drivers simply don't make sense here.


Alex

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
>>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>>>> +    reg.addr = (u64)&data;
>>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>>>> +
>>>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>>>>
>>>> This rules out guests where userspace and kernelspace can run with different
>>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>>>> userspace play with a virtio device...
>>>
>>> Yeah, I suppose we could check either EE or E0 depending on the mode
>>> when the access was made. We already have all the information, just need
>>> to handle the case. I'll respin the series.
>
>> How virtio implementations should determine their endianness is
>> a spec question, I think; at any rate QEMU and kvmtool ought to
>> agree on how it's done. I think the most recent suggestion on the
>> QEMU mailing list (for PPC) is that we should care about the
>> guest kernel endianness, but I don't know if anybody thought of
>> the pass-through-to-userspace usecase...
>
> Current opinion on the qemu-devel thread seems to be that we
> should just define that the endianness of the virtio device is
> the endianness of the guest kernel at the point where the guest
> triggers a reset of the virtio device by writing zero the QueuePFN
> or Status registers.

On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
going to simply explode if the access comes from userspace?

On AArch64, we can either select the kernel endianness, or userspace
endianness. Are we going to go a different route just for the sake of
enforcing kernel access?

I'm inclined to think of userspace access as a valid use case.

	M.
-- 
Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Alexander Graf]

> Am 07.05.2014 um 11:52 schrieb Marc Zyngier <marc.zyngier@arm.com>:
> 
>> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>>> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
>>>>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>>>>> +    reg.addr = (u64)&data;
>>>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>>>>> +
>>>>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>>>>> 
>>>>> This rules out guests where userspace and kernelspace can run with different
>>>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>>>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>>>>> userspace play with a virtio device...
>>>> 
>>>> Yeah, I suppose we could check either EE or E0 depending on the mode
>>>> when the access was made. We already have all the information, just need
>>>> to handle the case. I'll respin the series.
>> 
>>> How virtio implementations should determine their endianness is
>>> a spec question, I think; at any rate QEMU and kvmtool ought to
>>> agree on how it's done. I think the most recent suggestion on the
>>> QEMU mailing list (for PPC) is that we should care about the
>>> guest kernel endianness, but I don't know if anybody thought of
>>> the pass-through-to-userspace usecase...
>> 
>> Current opinion on the qemu-devel thread seems to be that we
>> should just define that the endianness of the virtio device is
>> the endianness of the guest kernel at the point where the guest
>> triggers a reset of the virtio device by writing zero the QueuePFN
>> or Status registers.
> 
> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
> going to simply explode if the access comes from userspace?
> 
> On AArch64, we can either select the kernel endianness, or userspace
> endianness. Are we going to go a different route just for the sake of
> enforcing kernel access?
> 
> I'm inclined to think of userspace access as a valid use case.

It's not for virtio-legacy. It'll be much more productive to influence virtio-1.0 to not redo the same mistakes than enabling even more hackery with the legacy one.

Alex

> 
>    M.
> -- 
> Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On Wed, May 07 2014 at 10:42:54 am BST, Alexander Graf <agraf@suse.de> wrote:
>> Am 07.05.2014 um 11:34 schrieb Peter Maydell <peter.maydell@linaro.org>:
>> 
>>> On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>>> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
>>>>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>>>>> +    reg.addr = (u64)&data;
>>>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>>>>> +
>>>>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>>>>> 
>>>>> This rules out guests where userspace and kernelspace can run with different
>>>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>>>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>>>>> userspace play with a virtio device...
>>>> 
>>>> Yeah, I suppose we could check either EE or E0 depending on the mode
>>>> when the access was made. We already have all the information, just need
>>>> to handle the case. I'll respin the series.
>> 
>>> How virtio implementations should determine their endianness is
>>> a spec question, I think; at any rate QEMU and kvmtool ought to
>>> agree on how it's done. I think the most recent suggestion on the
>>> QEMU mailing list (for PPC) is that we should care about the
>>> guest kernel endianness, but I don't know if anybody thought of
>>> the pass-through-to-userspace usecase...
>> 
>> Current opinion on the qemu-devel thread seems to be that we
>> should just define that the endianness of the virtio device is
>> the endianness of the guest kernel at the point where the guest
>> triggers a reset of the virtio device by writing zero the QueuePFN
>> or Status registers.
>
> Virtio by design has full access to guest physical memory. It doesn't
> route DMA via PCI. So user space drivers simply don't make sense here.

Huh? What if my guest has usespace using an idmap, with Stage-1 MMU for
isolation only (much like an MPU)? R-class guests anyone?

Agreed, this is not the general use case, but that doesn't seem to be
completely unrealistic either.

	M.
-- 
Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Peter Maydell]

On 7 May 2014 10:52, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell <peter.maydell@linaro.org> wrote:
>> Current opinion on the qemu-devel thread seems to be that we
>> should just define that the endianness of the virtio device is
>> the endianness of the guest kernel at the point where the guest
>> triggers a reset of the virtio device by writing zero the QueuePFN
>> or Status registers.
>
> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
> going to simply explode if the access comes from userspace?

There's SCTLR.EE in AArch32, right?

> On AArch64, we can either select the kernel endianness, or userspace
> endianness. Are we going to go a different route just for the sake of
> enforcing kernel access?
>
> I'm inclined to think of userspace access as a valid use case.

I don't actually care much about the details of what we decide;
I just want us to be consistent between QEMU and kvmtool and
(to the extent that architectural differences permit) consistent
between PPC and ARM. At the moment we seem to be heading
in gratuitously different directions.

thanks
-- PMM

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Alexander Graf]

On 05/07/2014 11:57 AM, Marc Zyngier wrote:
> On Wed, May 07 2014 at 10:42:54 am BST, Alexander Graf <agraf@suse.de> wrote:
>>> Am 07.05.2014 um 11:34 schrieb Peter Maydell <peter.maydell@linaro.org>:
>>>
>>>> On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>>> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>>>> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
>>>>>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>>>>>> +    reg.addr = (u64)&data;
>>>>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>>>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>>>>>> +
>>>>>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
>>>>>> This rules out guests where userspace and kernelspace can run with different
>>>>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>>>>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>>>>>> userspace play with a virtio device...
>>>>> Yeah, I suppose we could check either EE or E0 depending on the mode
>>>>> when the access was made. We already have all the information, just need
>>>>> to handle the case. I'll respin the series.
>>>> How virtio implementations should determine their endianness is
>>>> a spec question, I think; at any rate QEMU and kvmtool ought to
>>>> agree on how it's done. I think the most recent suggestion on the
>>>> QEMU mailing list (for PPC) is that we should care about the
>>>> guest kernel endianness, but I don't know if anybody thought of
>>>> the pass-through-to-userspace usecase...
>>> Current opinion on the qemu-devel thread seems to be that we
>>> should just define that the endianness of the virtio device is
>>> the endianness of the guest kernel at the point where the guest
>>> triggers a reset of the virtio device by writing zero the QueuePFN
>>> or Status registers.
>> Virtio by design has full access to guest physical memory. It doesn't
>> route DMA via PCI. So user space drivers simply don't make sense here.
> Huh? What if my guest has usespace using an idmap, with Stage-1 MMU for
> isolation only (much like an MPU)? R-class guests anyone?
>
> Agreed, this is not the general use case, but that doesn't seem to be
> completely unrealistic either.

Yes, and once that user tries the same without idmap virtio ends up 
overwriting random memory. It's just not a good idea and I'd much rather 
see us solve this properly with virtio 1.0 really.

Of course alternatively we can continue bikeshedding about this until 
everything becomes moot because we switched to virtio 1.0 ;).

Rusty / Michael, virtio 1.0 does go via normal DMA channels, right?


Alex

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On Wed, May 07 2014 at 10:55:45 am BST, Alexander Graf <agraf@suse.de> wrote:
>> Am 07.05.2014 um 11:52 schrieb Marc Zyngier <marc.zyngier@arm.com>:
>> 
>>> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell
>>> <peter.maydell@linaro.org> wrote:
>>>> On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>>> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>>>> On Tue, May 06 2014 at 3:28:07 pm BST, Will Deacon
>>>>>> <will.deacon@arm.com> wrote:
>>>>>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>>>>>>> +    reg.addr = (u64)&data;
>>>>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>>>>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>>>>>>> +
>>>>>>> + return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE :
>>>>>>> VIRTIO_ENDIAN_LE;
>>>>>> 
>>>>>> This rules out guests where userspace and kernelspace can run
>>>>>> with different
>>>>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>>>>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>>>>>> userspace play with a virtio device...
>>>>> 
>>>>> Yeah, I suppose we could check either EE or E0 depending on the mode
>>>>> when the access was made. We already have all the information, just need
>>>>> to handle the case. I'll respin the series.
>>> 
>>>> How virtio implementations should determine their endianness is
>>>> a spec question, I think; at any rate QEMU and kvmtool ought to
>>>> agree on how it's done. I think the most recent suggestion on the
>>>> QEMU mailing list (for PPC) is that we should care about the
>>>> guest kernel endianness, but I don't know if anybody thought of
>>>> the pass-through-to-userspace usecase...
>>> 
>>> Current opinion on the qemu-devel thread seems to be that we
>>> should just define that the endianness of the virtio device is
>>> the endianness of the guest kernel at the point where the guest
>>> triggers a reset of the virtio device by writing zero the QueuePFN
>>> or Status registers.
>> 
>> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
>> going to simply explode if the access comes from userspace?
>> 
>> On AArch64, we can either select the kernel endianness, or userspace
>> endianness. Are we going to go a different route just for the sake of
>> enforcing kernel access?
>> 
>> I'm inclined to think of userspace access as a valid use case.
>
> It's not for virtio-legacy. It'll be much more productive to influence
> virtio-1.0 to not redo the same mistakes than enabling even more
> hackery with the legacy one.

Are you saying I shouldn't improve an existing code base and implement a
useful feature, and should instead work on some new fancy stuff for
which there is no platform support, no kernel support, and not an
official spec either? Watch me.

	M.
-- 
Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Michael S. Tsirkin]

On Wed, May 07, 2014 at 12:11:13PM +0200, Alexander Graf wrote:
> On 05/07/2014 11:57 AM, Marc Zyngier wrote:
> >On Wed, May 07 2014 at 10:42:54 am BST, Alexander Graf <agraf@suse.de> wrote:
> >>>Am 07.05.2014 um 11:34 schrieb Peter Maydell <peter.maydell@linaro.org>:
> >>>
> >>>>On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
> >>>>>On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
> >>>>>>On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
> >>>>>>>On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
> >>>>>>>+    reg.addr = (u64)&data;
> >>>>>>>+    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
> >>>>>>>+            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
> >>>>>>>+
> >>>>>>>+    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
> >>>>>>This rules out guests where userspace and kernelspace can run with different
> >>>>>>endinness. Whilst Linux doesn't currently do this, can we support it here?
> >>>>>>It all gets a bit hairy if the guest is using a stage-1 SMMU to let
> >>>>>>userspace play with a virtio device...
> >>>>>Yeah, I suppose we could check either EE or E0 depending on the mode
> >>>>>when the access was made. We already have all the information, just need
> >>>>>to handle the case. I'll respin the series.
> >>>>How virtio implementations should determine their endianness is
> >>>>a spec question, I think; at any rate QEMU and kvmtool ought to
> >>>>agree on how it's done. I think the most recent suggestion on the
> >>>>QEMU mailing list (for PPC) is that we should care about the
> >>>>guest kernel endianness, but I don't know if anybody thought of
> >>>>the pass-through-to-userspace usecase...
> >>>Current opinion on the qemu-devel thread seems to be that we
> >>>should just define that the endianness of the virtio device is
> >>>the endianness of the guest kernel at the point where the guest
> >>>triggers a reset of the virtio device by writing zero the QueuePFN
> >>>or Status registers.
> >>Virtio by design has full access to guest physical memory. It doesn't
> >>route DMA via PCI. So user space drivers simply don't make sense here.
> >Huh? What if my guest has usespace using an idmap, with Stage-1 MMU for
> >isolation only (much like an MPU)? R-class guests anyone?
> >
> >Agreed, this is not the general use case, but that doesn't seem to be
> >completely unrealistic either.
> 
> Yes, and once that user tries the same without idmap virtio ends up
> overwriting random memory. It's just not a good idea and I'd much
> rather see us solve this properly with virtio 1.0 really.
> 
> Of course alternatively we can continue bikeshedding about this
> until everything becomes moot because we switched to virtio 1.0 ;).
> 
> Rusty / Michael, virtio 1.0 does go via normal DMA channels, right?
> 
> 
> Alex

By default it doesn't at the moment, in particular IOMMUs really
seem to hurt performance when enabled, and many guests seem to be
dumb and enable it everywhere if present.

By design IOMMUs can protect you from malicious devices, which
is relevant if you assign a device but of course isn't
relevant for qemu as virtio is part of qemu atm.


In virtio 1.0 it's possible for a device to have "required features"
which drivers must ack.
So we'll be able to have hypervisor tell guest that it requires DMA for
some VFs.
You would then be able to do mix fast virtio PF bypassing IOMMU and handle
that in kernel, and slow virtio VF going through an IOMMU and handle that
in userspace.

-- 
MST

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On Wed, May 07 2014 at 11:11:13 am BST, Alexander Graf <agraf@suse.de> wrote:
> On 05/07/2014 11:57 AM, Marc Zyngier wrote:

>> Huh? What if my guest has usespace using an idmap, with Stage-1 MMU for
>> isolation only (much like an MPU)? R-class guests anyone?
>>
>> Agreed, this is not the general use case, but that doesn't seem to be
>> completely unrealistic either.
>
> Yes, and once that user tries the same without idmap virtio ends up
> overwriting random memory.

And how different is that from the kernel suddenly deciding to use VAs
instead of PAs? Just as broken. Are we going to prevent the kernel from
using virtio?

> It's just not a good idea and I'd much rather see us solve this
> properly with virtio 1.0 really.

Again, what is virtio 1.0 doing here?

       M.
-- 
Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Greg Kurz]

On Wed, 07 May 2014 10:52:01 +0100
Marc Zyngier <marc.zyngier@arm.com> wrote:

> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell <peter.maydell@linaro.org> wrote:
> > On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
> >> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
> >>> On Tue, May 06 2014 at  3:28:07 pm BST, Will Deacon <will.deacon@arm.com> wrote:
> >>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
> >>>>> +    reg.addr = (u64)&data;
> >>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
> >>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
> >>>>> +
> >>>>> +    return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE : VIRTIO_ENDIAN_LE;
> >>>>
> >>>> This rules out guests where userspace and kernelspace can run with different
> >>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
> >>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
> >>>> userspace play with a virtio device...
> >>>
> >>> Yeah, I suppose we could check either EE or E0 depending on the mode
> >>> when the access was made. We already have all the information, just need
> >>> to handle the case. I'll respin the series.
> >
> >> How virtio implementations should determine their endianness is
> >> a spec question, I think; at any rate QEMU and kvmtool ought to
> >> agree on how it's done. I think the most recent suggestion on the
> >> QEMU mailing list (for PPC) is that we should care about the
> >> guest kernel endianness, but I don't know if anybody thought of
> >> the pass-through-to-userspace usecase...
> >
> > Current opinion on the qemu-devel thread seems to be that we
> > should just define that the endianness of the virtio device is
> > the endianness of the guest kernel at the point where the guest
> > triggers a reset of the virtio device by writing zero the QueuePFN
> > or Status registers.
> 
> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
> going to simply explode if the access comes from userspace?
> 
> On AArch64, we can either select the kernel endianness, or userspace
> endianness. Are we going to go a different route just for the sake of
> enforcing kernel access?
> 
> I'm inclined to think of userspace access as a valid use case.
> 
> 	M.

All the fuzz is not really about enforcing kernel access... PPC also
has a current endianness selector (MSR_LE) but it only makes sense
if you are in the cpu context. Initial versions of the virtio biendian
support for QEMU PPC64 used an arbitrary cpu: in this case, the
only sensible thing to look at to support kernel based virtio is the 
interrupt endianness selector (LPCR_ILE), because if gives a safe
hint of the kernel endianness.

The patch set has evolved and now uses current_cpu at device reset time.
As a consequence, we are not necessarily tied to the kernel LPCR_ILE
selector I guess.

Cheers.

-- 
Gregory Kurz                                     kurzgreg@fr.ibm.com
                                                 gkurz@linux.vnet.ibm.com
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On Wed, May 07 2014 at 11:10:56 am BST, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 7 May 2014 10:52, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell
>> <peter.maydell@linaro.org> wrote:
>>> Current opinion on the qemu-devel thread seems to be that we
>>> should just define that the endianness of the virtio device is
>>> the endianness of the guest kernel at the point where the guest
>>> triggers a reset of the virtio device by writing zero the QueuePFN
>>> or Status registers.
>>
>> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
>> going to simply explode if the access comes from userspace?
>
> There's SCTLR.EE in AArch32, right?

Indeed, good point.

>> On AArch64, we can either select the kernel endianness, or userspace
>> endianness. Are we going to go a different route just for the sake of
>> enforcing kernel access?
>>
>> I'm inclined to think of userspace access as a valid use case.
>
> I don't actually care much about the details of what we decide; I just
> want us to be consistent between QEMU and kvmtool and (to the extent
> that architectural differences permit) consistent between PPC and
> ARM. At the moment we seem to be heading in gratuitously different
> directions.

My point is: is there any good technical reason for deciding not to
support guest user space access, other than religious matters about the
latest incarnation of The Holy Virtio Spec?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On Wed, May 07 2014 at 11:40:54 am BST, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
> On Wed, 07 May 2014 10:52:01 +0100
> Marc Zyngier <marc.zyngier@arm.com> wrote:
>
>> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell
>> <peter.maydell@linaro.org> wrote:
>> > On 6 May 2014 19:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>> >> On 6 May 2014 18:25, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> >>> On Tue, May 06 2014 at 3:28:07 pm BST, Will Deacon
>> >>> <will.deacon@arm.com> wrote:
>> >>>> On Thu, Apr 24, 2014 at 07:17:23PM +0100, Marc Zyngier wrote:
>> >>>>> +    reg.addr = (u64)&data;
>> >>>>> +    if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
>> >>>>> +            die("KVM_GET_ONE_REG failed (SCTLR_EL1)");
>> >>>>> +
>> >>>>> + return (data & SCTLR_EL1_EE_MASK) ? VIRTIO_ENDIAN_BE :
>> >>>>> VIRTIO_ENDIAN_LE;
>> >>>>
>> >>>> This rules out guests where userspace and kernelspace can run
>> >>>> with different
>> >>>> endinness. Whilst Linux doesn't currently do this, can we support it here?
>> >>>> It all gets a bit hairy if the guest is using a stage-1 SMMU to let
>> >>>> userspace play with a virtio device...
>> >>>
>> >>> Yeah, I suppose we could check either EE or E0 depending on the mode
>> >>> when the access was made. We already have all the information, just need
>> >>> to handle the case. I'll respin the series.
>> >
>> >> How virtio implementations should determine their endianness is
>> >> a spec question, I think; at any rate QEMU and kvmtool ought to
>> >> agree on how it's done. I think the most recent suggestion on the
>> >> QEMU mailing list (for PPC) is that we should care about the
>> >> guest kernel endianness, but I don't know if anybody thought of
>> >> the pass-through-to-userspace usecase...
>> >
>> > Current opinion on the qemu-devel thread seems to be that we
>> > should just define that the endianness of the virtio device is
>> > the endianness of the guest kernel at the point where the guest
>> > triggers a reset of the virtio device by writing zero the QueuePFN
>> > or Status registers.
>> 
>> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
>> going to simply explode if the access comes from userspace?
>> 
>> On AArch64, we can either select the kernel endianness, or userspace
>> endianness. Are we going to go a different route just for the sake of
>> enforcing kernel access?
>> 
>> I'm inclined to think of userspace access as a valid use case.
>> 
>> 	M.

Hi Gregory,

> All the fuzz is not really about enforcing kernel access... PPC also
> has a current endianness selector (MSR_LE) but it only makes sense
> if you are in the cpu context. Initial versions of the virtio biendian
> support for QEMU PPC64 used an arbitrary cpu: in this case, the
> only sensible thing to look at to support kernel based virtio is the 
> interrupt endianness selector (LPCR_ILE), because if gives a safe
> hint of the kernel endianness.
>
> The patch set has evolved and now uses current_cpu at device reset time.
> As a consequence, we are not necessarily tied to the kernel LPCR_ILE
> selector I guess.

That makes a lot of sense, thanks for explaining that. You're basically
doing the exact same thing we do with kvmtool on ARM. So if we have
similar architectural features on both sides, why don't we support both
kernel and userspace access?

Cheers,

	M.
-- 
Jazz is not dead. It just smells funny.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Alexander Graf]

On 05/07/2014 12:46 PM, Marc Zyngier wrote:
> On Wed, May 07 2014 at 11:10:56 am BST, Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 7 May 2014 10:52, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell
>>> <peter.maydell@linaro.org> wrote:
>>>> Current opinion on the qemu-devel thread seems to be that we
>>>> should just define that the endianness of the virtio device is
>>>> the endianness of the guest kernel at the point where the guest
>>>> triggers a reset of the virtio device by writing zero the QueuePFN
>>>> or Status registers.
>>> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
>>> going to simply explode if the access comes from userspace?
>> There's SCTLR.EE in AArch32, right?
> Indeed, good point.
>
>>> On AArch64, we can either select the kernel endianness, or userspace
>>> endianness. Are we going to go a different route just for the sake of
>>> enforcing kernel access?
>>>
>>> I'm inclined to think of userspace access as a valid use case.
>> I don't actually care much about the details of what we decide; I just
>> want us to be consistent between QEMU and kvmtool and (to the extent
>> that architectural differences permit) consistent between PPC and
>> ARM. At the moment we seem to be heading in gratuitously different
>> directions.
> My point is: is there any good technical reason for deciding not to
> support guest user space access, other than religious matters about the
> latest incarnation of The Holy Virtio Spec?

Yes, because it can't be isolated as per the current spec. User space 
has no business in physical addresses. And since so far I haven't heard 
of a single case where people on ARM are either

   a) nesting virtualization or
   b) running different endian user space

I don't think this point is valid. Virtio 1.0 is defined to be little 
endian only, so we don't need all that messy magic logic anymore. By the 
time people will do nesting or different endian user space we will most 
likely be in virtio 1.0 land. Shoehorning in anything in between is just 
a waste of time.

If you like to see a constructed case where your logic falls apart, I 
can easily give you one too (because the whole thing is just insanely 
fragile). Imagine you have nesting. Your L1 guest passes its virtio 
device into the L2 guest with idmap. The L1 guest wants to trace MMIO 
accesses, so it traps on every access and delivers it on its own. L2 is 
LE, L1 is BE. Virtio gets initialized BE even through the guest that 
really wants to access it is LE.


Alex

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On 07/05/14 12:49, Alexander Graf wrote:
> On 05/07/2014 12:46 PM, Marc Zyngier wrote:
>> On Wed, May 07 2014 at 11:10:56 am BST, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> On 7 May 2014 10:52, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>> On Wed, May 07 2014 at 10:34:30 am BST, Peter Maydell
>>>> <peter.maydell@linaro.org> wrote:
>>>>> Current opinion on the qemu-devel thread seems to be that we
>>>>> should just define that the endianness of the virtio device is
>>>>> the endianness of the guest kernel at the point where the guest
>>>>> triggers a reset of the virtio device by writing zero the QueuePFN
>>>>> or Status registers.
>>>> On AArch32, we only have the CPSR.E bit to select the endiannes. Are we
>>>> going to simply explode if the access comes from userspace?
>>> There's SCTLR.EE in AArch32, right?
>> Indeed, good point.
>>
>>>> On AArch64, we can either select the kernel endianness, or userspace
>>>> endianness. Are we going to go a different route just for the sake of
>>>> enforcing kernel access?
>>>>
>>>> I'm inclined to think of userspace access as a valid use case.
>>> I don't actually care much about the details of what we decide; I just
>>> want us to be consistent between QEMU and kvmtool and (to the extent
>>> that architectural differences permit) consistent between PPC and
>>> ARM. At the moment we seem to be heading in gratuitously different
>>> directions.
>> My point is: is there any good technical reason for deciding not to
>> support guest user space access, other than religious matters about the
>> latest incarnation of The Holy Virtio Spec?
> 
> Yes, because it can't be isolated as per the current spec. User space 
> has no business in physical addresses. And since so far I haven't heard 
> of a single case where people on ARM are either
> 
>    a) nesting virtualization or
>    b) running different endian user space
> 
> I don't think this point is valid. Virtio 1.0 is defined to be little 
> endian only, so we don't need all that messy magic logic anymore. By the 

Alex, please read my lips: at the moment, I don't care about virtio-1.0.
At all. Doesn't register. And hammering it on and on won't change a
thing (yes, I've rewritten this sentence at least five times to remove
all the fscking swear words).

> time people will do nesting or different endian user space we will most 
> likely be in virtio 1.0 land. Shoehorning in anything in between is just 
> a waste of time.

If you don't want to support it on your pet platform/environment, fine.

> If you like to see a constructed case where your logic falls apart, I 
> can easily give you one too (because the whole thing is just insanely 
> fragile). Imagine you have nesting. Your L1 guest passes its virtio 
> device into the L2 guest with idmap. The L1 guest wants to trace MMIO 
> accesses, so it traps on every access and delivers it on its own. L2 is 
> LE, L1 is BE. Virtio gets initialized BE even through the guest that 
> really wants to access it is LE.

Then it is a bug in your L1 that doesn't properly emulate accesses it
traps. Not that I care, really.

That being said, I'm going to stop replying to this thread, and instead
go back writing code, posting it, and getting on with my life in
virtio-legacy land.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Peter Maydell]

On 7 May 2014 12:04, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On Wed, May 07 2014 at 11:40:54 am BST, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
>> All the fuzz is not really about enforcing kernel access... PPC also
>> has a current endianness selector (MSR_LE) but it only makes sense
>> if you are in the cpu context. Initial versions of the virtio biendian
>> support for QEMU PPC64 used an arbitrary cpu: in this case, the
>> only sensible thing to look at to support kernel based virtio is the
>> interrupt endianness selector (LPCR_ILE), because if gives a safe
>> hint of the kernel endianness.
>>
>> The patch set has evolved and now uses current_cpu at device reset time.
>> As a consequence, we are not necessarily tied to the kernel LPCR_ILE
>> selector I guess.

Ah yes, I'd forgotten the history behind why we ended up looking
at interrupt endianness.

> That makes a lot of sense, thanks for explaining that. You're basically
> doing the exact same thing we do with kvmtool on ARM. So if we have
> similar architectural features on both sides, why don't we support both
> kernel and userspace access?

I don't think that we really need to get into whether userspace
access is or is not a good idea -- "endianness of the CPU which
does the virtio reset at the point when it does that reset" is a
nice simple rule that should generalise across architectures,
so why make it more complicated than that?

thanks
-- PMM

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Peter Maydell]

On 7 May 2014 13:16, Marc Zyngier <marc.zyngier@arm.com> wrote:
> That being said, I'm going to stop replying to this thread, and instead
> go back writing code, posting it, and getting on with my life in
> virtio-legacy land.

Some of us are trying to have a conversation in this thread
about virtio-legacy behaviour :-)

thanks
-- PMM

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Marc Zyngier]

On 07/05/14 13:17, Peter Maydell wrote:
> On 7 May 2014 12:04, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> On Wed, May 07 2014 at 11:40:54 am BST, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
>>> All the fuzz is not really about enforcing kernel access... PPC also
>>> has a current endianness selector (MSR_LE) but it only makes sense
>>> if you are in the cpu context. Initial versions of the virtio biendian
>>> support for QEMU PPC64 used an arbitrary cpu: in this case, the
>>> only sensible thing to look at to support kernel based virtio is the
>>> interrupt endianness selector (LPCR_ILE), because if gives a safe
>>> hint of the kernel endianness.
>>>
>>> The patch set has evolved and now uses current_cpu at device reset time.
>>> As a consequence, we are not necessarily tied to the kernel LPCR_ILE
>>> selector I guess.
> 
> Ah yes, I'd forgotten the history behind why we ended up looking
> at interrupt endianness.
> 
>> That makes a lot of sense, thanks for explaining that. You're basically
>> doing the exact same thing we do with kvmtool on ARM. So if we have
>> similar architectural features on both sides, why don't we support both
>> kernel and userspace access?
> 
> I don't think that we really need to get into whether userspace
> access is or is not a good idea -- "endianness of the CPU which
> does the virtio reset at the point when it does that reset" is a
> nice simple rule that should generalise across architectures,
> so why make it more complicated than that?

This definition looks pretty good to me. Simple and to the point.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Greg Kurz]

On Wed, 7 May 2014 13:17:51 +0100
Peter Maydell <peter.maydell@linaro.org> wrote:

> On 7 May 2014 12:04, Marc Zyngier <marc.zyngier@arm.com> wrote:
> > On Wed, May 07 2014 at 11:40:54 am BST, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
> >> All the fuzz is not really about enforcing kernel access... PPC also
> >> has a current endianness selector (MSR_LE) but it only makes sense
> >> if you are in the cpu context. Initial versions of the virtio biendian
> >> support for QEMU PPC64 used an arbitrary cpu: in this case, the
> >> only sensible thing to look at to support kernel based virtio is the
> >> interrupt endianness selector (LPCR_ILE), because if gives a safe
> >> hint of the kernel endianness.
> >>
> >> The patch set has evolved and now uses current_cpu at device reset time.
> >> As a consequence, we are not necessarily tied to the kernel LPCR_ILE
> >> selector I guess.
> 
> Ah yes, I'd forgotten the history behind why we ended up looking
> at interrupt endianness.
> 
> > That makes a lot of sense, thanks for explaining that. You're basically
> > doing the exact same thing we do with kvmtool on ARM. So if we have
> > similar architectural features on both sides, why don't we support both
> > kernel and userspace access?
> 
> I don't think that we really need to get into whether userspace
> access is or is not a good idea -- "endianness of the CPU which
> does the virtio reset at the point when it does that reset" is a
> nice simple rule that should generalise across architectures,
> so why make it more complicated than that?
> 
> thanks
> -- PMM
> 

I am convinced... and feeling a bit guilty for all the noise ;)
I'll come with a new virtio patch set for QEMU that does just what
you say.

-- 
Gregory Kurz                                     kurzgreg@fr.ibm.com
                                                 gkurz@linux.vnet.ibm.com
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.

Re: [PATCH v3 9/9] kvmtool: virtio: enable arm/arm64 support for bi-endianness

[Rusty Russell]

Alexander Graf <agraf@suse.de> writes:
> On 05/07/2014 11:57 AM, Marc Zyngier wrote:
>>>>> How virtio implementations should determine their endianness is
>>>>> a spec question, I think; at any rate QEMU and kvmtool ought to
>>>>> agree on how it's done. I think the most recent suggestion on the
>>>>> QEMU mailing list (for PPC) is that we should care about the
>>>>> guest kernel endianness, but I don't know if anybody thought of
>>>>> the pass-through-to-userspace usecase...
>>>> Current opinion on the qemu-devel thread seems to be that we
>>>> should just define that the endianness of the virtio device is
>>>> the endianness of the guest kernel at the point where the guest
>>>> triggers a reset of the virtio device by writing zero the QueuePFN
>>>> or Status registers.
>>> Virtio by design has full access to guest physical memory. It doesn't
>>> route DMA via PCI. So user space drivers simply don't make sense here.
>> Huh? What if my guest has usespace using an idmap, with Stage-1 MMU for
>> isolation only (much like an MPU)? R-class guests anyone?
>>
>> Agreed, this is not the general use case, but that doesn't seem to be
>> completely unrealistic either.
>
> Yes, and once that user tries the same without idmap virtio ends up 
> overwriting random memory. It's just not a good idea and I'd much rather 
> see us solve this properly with virtio 1.0 really.

Slightly orthogonal: virtio 1.0 is LE, so endianness is solved.

> Of course alternatively we can continue bikeshedding about this until 
> everything becomes moot because we switched to virtio 1.0 ;).

Transition will be long...

> Rusty / Michael, virtio 1.0 does go via normal DMA channels, right?

No.  We argued about this; it's more PCI-like to do, but there's a
performance cost and it's really unclear that passing through a virtio
PCI device to a (sub)guest is a scenario worth supporting.

Maybe someone will come up with a convincing reason, and we'll add
a feature bit in a future revision...

Cheers,
Rusty.