[PATCH V4 0/2] introduce coalesced pio support

[PATCH V4 0/2] introduce coalesced pio support

[Peng Hao]

Coalesced pio is base on coalesced mmio and can be used for some port
like rtc port, pci-host config port, virtio-pci config port and so on.

Specially in case of rtc as coalesced pio, some versions of windows guest
access rtc frequently because of rtc as system tick. guest access rtc like
this: write register index to 0x70, then write or read data from 0x71.
writing 0x70 port is just as index and do nothing else. So we can use
coalesced pio to handle this scene to reduce VM-EXIT time.

When it starts and closes the virtual machine, it will access pci-host config
port or virtio-pci config port frequently. So setting these port as
coalesced pio can reduce startup and shutdown time. In qemu I just realize
piixfx's pci-host, it is convenient for other pci-host type implementations.

without my patch, get the vm-exit time of accessing rtc 0x70 and piix 0xcf8
using perf tools: (guest OS : windows 7 64bit)
IO Port Access  Samples Samples%  Time%  Min Time  Max Time  Avg time
0x70:POUT        86     30.99%    74.59%   9us      29us    10.75us (+- 3.41%)
0xcf8:POUT     1119     2.60%     2.12%   2.79us    56.83us 3.41us (+- 2.23%)

with my patch
IO Port Access  Samples Samples%  Time%   Min Time  Max Time   Avg time
0x70:POUT       106    32.02%    29.47%    0us      10us     1.57us (+- 7.38%)
0xcf8:POUT      1065    1.67%     0.28%   0.41us    65.44us   0.66us (+- 10.55%)


Peng Hao (2):
  kvm/x86 : add coalesced pio support
  kvm/x86 : add document for coalesced pio

 Documentation/virtual/kvm/00-INDEX          |  2 ++
 Documentation/virtual/kvm/coalesced-pio.txt | 15 ++++++++++++++
 include/uapi/linux/kvm.h                    | 11 +++++++++--
 virt/kvm/coalesced_mmio.c                   | 12 +++++++++---
 virt/kvm/kvm_main.c                         |  2 ++
 5 files changed, 37 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/virtual/kvm/coalesced-pio.txt

-- 
1.8.3.1

[PATCH V4 1/2] kvm/x86 : add coalesced pio support

[Peng Hao]

Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
---
 include/uapi/linux/kvm.h  | 11 +++++++++--
 virt/kvm/coalesced_mmio.c | 12 +++++++++---
 virt/kvm/kvm_main.c       |  2 ++
 3 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index b6270a3..a44e08e 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -420,13 +420,19 @@ struct kvm_run {
 struct kvm_coalesced_mmio_zone {
 	__u64 addr;
 	__u32 size;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 pio;
+	};
 };
 
 struct kvm_coalesced_mmio {
 	__u64 phys_addr;
 	__u32 len;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 pio;
+	};
 	__u8  data[8];
 };
 
@@ -949,6 +955,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_GET_MSR_FEATURES 153
 #define KVM_CAP_HYPERV_EVENTFD 154
 #define KVM_CAP_HYPERV_TLBFLUSH 155
+#define KVM_CAP_COALESCED_PIO 156
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff --git a/virt/kvm/coalesced_mmio.c b/virt/kvm/coalesced_mmio.c
index 9e65feb..cf12825 100644
--- a/virt/kvm/coalesced_mmio.c
+++ b/virt/kvm/coalesced_mmio.c
@@ -83,6 +83,7 @@ static int coalesced_mmio_write(struct kvm_vcpu *vcpu,
 	ring->coalesced_mmio[ring->last].phys_addr = addr;
 	ring->coalesced_mmio[ring->last].len = len;
 	memcpy(ring->coalesced_mmio[ring->last].data, val, len);
+	ring->coalesced_mmio[ring->last].pio = dev->zone.pio;
 	smp_wmb();
 	ring->last = (ring->last + 1) % KVM_COALESCED_MMIO_MAX;
 	spin_unlock(&dev->kvm->ring_lock);
@@ -140,6 +141,9 @@ int kvm_vm_ioctl_register_coalesced_mmio(struct kvm *kvm,
 	int ret;
 	struct kvm_coalesced_mmio_dev *dev;
 
+	if (zone->pio != 1 && zone->pio != 0)
+		return -EINVAL;
+
 	dev = kzalloc(sizeof(struct kvm_coalesced_mmio_dev), GFP_KERNEL);
 	if (!dev)
 		return -ENOMEM;
@@ -149,8 +153,9 @@ int kvm_vm_ioctl_register_coalesced_mmio(struct kvm *kvm,
 	dev->zone = *zone;
 
 	mutex_lock(&kvm->slots_lock);
-	ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, zone->addr,
-				      zone->size, &dev->dev);
+	ret = kvm_io_bus_register_dev(kvm,
+				zone->pio ? KVM_PIO_BUS : KVM_MMIO_BUS,
+				zone->addr, zone->size, &dev->dev);
 	if (ret < 0)
 		goto out_free_dev;
 	list_add_tail(&dev->list, &kvm->coalesced_zones);
@@ -174,7 +179,8 @@ int kvm_vm_ioctl_unregister_coalesced_mmio(struct kvm *kvm,
 
 	list_for_each_entry_safe(dev, tmp, &kvm->coalesced_zones, list)
 		if (coalesced_mmio_in_range(dev, zone->addr, zone->size)) {
-			kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS, &dev->dev);
+			kvm_io_bus_unregister_dev(kvm,
+				zone->pio ? KVM_PIO_BUS : KVM_MMIO_BUS, &dev->dev);
 			kvm_iodevice_destructor(&dev->dev);
 		}
 
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 8b47507f..a587fb9 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2936,6 +2936,8 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 #ifdef CONFIG_KVM_MMIO
 	case KVM_CAP_COALESCED_MMIO:
 		return KVM_COALESCED_MMIO_PAGE_OFFSET;
+	case KVM_CAP_COALESCED_PIO:
+		return 1;
 #endif
 #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING
 	case KVM_CAP_IRQ_ROUTING:
-- 
1.8.3.1

[PATCH V4 2/2] kvm/x86 : add document for coalesced pio

[Peng Hao]

Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
---
 Documentation/virtual/kvm/00-INDEX          |  2 ++
 Documentation/virtual/kvm/coalesced-pio.txt | 15 +++++++++++++++
 2 files changed, 17 insertions(+)
 create mode 100644 Documentation/virtual/kvm/coalesced-pio.txt

diff --git a/Documentation/virtual/kvm/00-INDEX b/Documentation/virtual/kvm/00-INDEX
index 3492458..4b7a37b 100644
--- a/Documentation/virtual/kvm/00-INDEX
+++ b/Documentation/virtual/kvm/00-INDEX
@@ -9,6 +9,8 @@ arm
 	- internal ABI between the kernel and HYP (for arm/arm64)
 cpuid.txt
 	- KVM-specific cpuid leaves (x86).
+coalesced-pio.txt
+	- KVM_CAP_COALESCED_PIO
 devices/
 	- KVM_CAP_DEVICE_CTRL userspace API.
 halt-polling.txt
diff --git a/Documentation/virtual/kvm/coalesced-pio.txt b/Documentation/virtual/kvm/coalesced-pio.txt
new file mode 100644
index 0000000..38dac11
--- /dev/null
+++ b/Documentation/virtual/kvm/coalesced-pio.txt
@@ -0,0 +1,15 @@
+Linux KVM Coalesced PIO:
+============================
+Coalesced pio is base on coalesced mmio. When the write access to a port
+of a device does a simple work (just like setting a register ID or address
+index), then we can use coalesced pio.
+
+Setting a port as coalesced pio, the write access to the port don't need to
+exit to userspace, and just record the value in shared coalesced ring in kernel.
+Then following access to another port of the same device which relys on the
+first port's setting or a read access to the first port will set the previous
+write value of the first port firstly in usersapce according to the shared
+coalesced ring.
+
+Coalesced pio can be used for rtc 0x70 port, pci-host config port, virtio-pci
+config port and so on.
-- 
1.8.3.1

Re: [PATCH V4 2/2] kvm/x86 : add document for coalesced pio

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3156 bytes --]

On Fri, Aug 31, 2018 at 12:17:21AM +0800, Peng Hao wrote:
> Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> ---
>  Documentation/virtual/kvm/00-INDEX          |  2 ++
>  Documentation/virtual/kvm/coalesced-pio.txt | 15 +++++++++++++++
>  2 files changed, 17 insertions(+)
>  create mode 100644 Documentation/virtual/kvm/coalesced-pio.txt
> 
> diff --git a/Documentation/virtual/kvm/00-INDEX b/Documentation/virtual/kvm/00-INDEX
> index 3492458..4b7a37b 100644
> --- a/Documentation/virtual/kvm/00-INDEX
> +++ b/Documentation/virtual/kvm/00-INDEX
> @@ -9,6 +9,8 @@ arm
>  	- internal ABI between the kernel and HYP (for arm/arm64)
>  cpuid.txt
>  	- KVM-specific cpuid leaves (x86).
> +coalesced-pio.txt
> +	- KVM_CAP_COALESCED_PIO

Please document KVM_(UN)REGISTER_COALESCED_MMIO in api.txt instead of
creating a new file.  This is part of the ioctl API and it should be
documented there.

The first patch could document the existing
KVM_(UN)REGISTER_COALESCED_MMIO ioctls and then a second patch could add
PIO.

> diff --git a/Documentation/virtual/kvm/coalesced-pio.txt b/Documentation/virtual/kvm/coalesced-pio.txt
> new file mode 100644
> index 0000000..38dac11
> --- /dev/null
> +++ b/Documentation/virtual/kvm/coalesced-pio.txt
> @@ -0,0 +1,15 @@
> +Linux KVM Coalesced PIO:
> +============================
> +Coalesced pio is base on coalesced mmio. When the write access to a port

s/base/based/

> +of a device does a simple work (just like setting a register ID or address
> +index), then we can use coalesced pio.

"Simple work" is vague and does not describe when or when not to use
coalesced I/O.

I think the point of coalesced I/O is:

  Coalesced I/O is a performance optimization that defers hardware
  register write emulation so that userspace exits are avoided.  It is
  typically used to reduce the overhead of emulating frequently accessed
  hardware registers.

(and then your next paragraph below explains how it works)

> +
> +Setting a port as coalesced pio, the write access to the port don't need to
> +exit to userspace, and just record the value in shared coalesced ring in kernel.

Minor tweaks to the wording and s/port/hardware register/ so this can be
used to document both PIO and MMIO:

  When a hardware register is configured for coalesced I/O, write accesses
  do not exit to userspace and their value is recorded in a shared
  coalesced ring in the kernel.

> +Then following access to another port of the same device which relys on the
> +first port's setting or a read access to the first port will set the previous
> +write value of the first port firstly in usersapce according to the shared
> +coalesced ring.

Similar tweaks:

  A read access to the hardware register or a write access to another
  hardware register on the same device which relies on the first hardware
  register's value will cause the shared coalesced ring to be processed by
  userspace before emulating the current access.

> +
> +Coalesced pio can be used for rtc 0x70 port, pci-host config port, virtio-pci
> +config port and so on.
> -- 
> 1.8.3.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]