Re: [Qemu-devel] [Bug 1483070] [NEW] VIRTIO Sequential Write IOPS limits not working

From: Stefan Hajnoczi <stefanha@gmail.com>
To: Bug 1483070 <1483070@bugs.launchpad.net>
Cc: Kevin Wolf <kwolf@redhat.com>, Peter Lieven <pl@kamp.de>,
	berto@igalia.com, qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] [Bug 1483070] [NEW] VIRTIO Sequential Write IOPS limits not working
Date: Mon, 10 Aug 2015 10:59:00 +0100	[thread overview]
Message-ID: <20150810095900.GB31433@stefanha-thinkpad.redhat.com> (raw)
In-Reply-To: <20150810001525.6231.66973.malonedeb@gac.canonical.com>

[-- Attachment #1: Type: text/plain, Size: 2011 bytes --]

On Mon, Aug 10, 2015 at 12:15:25AM -0000, James Watson wrote:
> IOPS limit does not work for VIRTIO devices if the disk workload is a sequential write.
> 
> To confirm:
> IDE disk devices - the IOPS limit works fine. Disk transfer speed limit works fine.
> VIRTIO disk devices - the IOPS limit works fine for random IO (write/read) and sequential read, but not for sequential write. Disk transfer speed limits work fine.
> 
> Tested on Windows 7,10 and 2k12 (Fedora drivers used and here is the twist):
> virtio-win-0.1.96 (stable) or older won't limit write IO if workload is sequential.
> virtio-win-0.1.105 (latest) or newer will limit but I have had two test machines crash when under high workload using IOPS limit.
> 
> For Linux:
> The issue is also apparent, tested on Ubuntu 14.04
> 
> On the hypervisor (using KVM) machine I have tried with Qemu 2.1.2
> (3.16.0-4-amd64 - Debian 8) and Qemu 2.3.0 (3.19.8-1-pve - Proxmox 3.4
> and 4) using multiple machines but all are 64bit intel.
> 
> Even though the latest VIRTIO guest drivers fix the problem, the guest
> drivers shouldn't be able to ignore the limits the host puts in place or
> am I missing something??

This is probably due to I/O request merging:

Your benchmark application may generate 32 x 4KB write requests, but
they are merged by the virtio-blk device into just 1 x 128KB write
request.

The merging can happen inside the guest, depending on your benchmark
application and the guest kernel's I/O stack.  It also happens in QEMU's
virtio-blk emulation.

The most recent versions of QEMU merge both read and write requests.
Older versions only merged write requests.

It would be more intuitive for request merging to happen after QEMU I/O
throttling checks.  Currently QEMU's I/O queue plug/unplug isn't
advanced enough to do the request merging, so it's done earlier in the
virtio-blk emulation code.

I've CCed Kevin Wolf, Alberto Garcia, and Peter Lieven who may have more
thoughts on this.

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]