From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55010) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YhdU1-0004Li-Ek for qemu-devel@nongnu.org; Mon, 13 Apr 2015 08:28:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YhdTy-0004kF-8V for qemu-devel@nongnu.org; Mon, 13 Apr 2015 08:28:57 -0400 Received: from mail-lb0-x22f.google.com ([2a00:1450:4010:c04::22f]:33349) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YhdTx-0004hA-R1 for qemu-devel@nongnu.org; Mon, 13 Apr 2015 08:28:54 -0400 Received: by lbbzk7 with SMTP id zk7so57241386lbb.0 for ; Mon, 13 Apr 2015 05:28:52 -0700 (PDT) Message-ID: <552BB682.6070707@gmail.com> Date: Mon, 13 Apr 2015 14:28:50 +0200 From: ein MIME-Version: 1.0 References: <552834C1.9070105@gmail.com> <20150413014546.GB14218@ad.nay.redhat.com> In-Reply-To: <20150413014546.GB14218@ad.nay.redhat.com> Content-Type: multipart/mixed; boundary="------------060504050006020300040305" Subject: Re: [Qemu-devel] Very poor IO performance which looks like some design problem. List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: qemu-devel@nongnu.org This is a multi-part message in MIME format. --------------060504050006020300040305 Content-Type: multipart/alternative; boundary="------------060200000403050204070502" --------------060200000403050204070502 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Dear Fam, Check out my update please: http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg01318.html Using aio=3Dnative,cache=3Dnone results in 500%-2000% performance drop comparing to bare metal and 300%-1000% comparing to aio=3Dthreads,cache=3Dunsafe. I'd like to know what numbers should I expect. Can somebody share what results do you have with aio=3Dnative in sequential IO ops? And of course= please add some info about disk and controller configuration. Maybe there's some bug in current version of Qemu in Debian 8 - testing. Package details: https://packages.debian.org/jessie/qemu-kvm . Version: *qemu-kvm (1:2.1+dfsg-11) * On 04/13/2015 03:45 AM, Fam Zheng wrote: > On Fri, 04/10 22:38, ein wrote: >> Qemu creates more than 70 threads and everyone of them tries to write = to >> disk, which results in: >> 1. High I/O time. >> 2. Large latency. >> 2. Poor sequential read/write speeds. >> >> When I limited number of cores, I guess I limited number of threads as= >> well. That's why I got better numbers. >> >> I've tried to combine AIO native and thread setting with deadline >> scheduler. Native AIO was much more worse. >> >> The final question, is there any way to prevent Qemu for making so lar= ge >> number of processes when VM does only one sequential R/W operation? > aio=3Dnative will make QEMU only submit IO from the IO thread, so you s= houldn't > see 70 threads with that. And that should usually be a better option fo= r > performance. > > > Fam --------------060200000403050204070502 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit Dear Fam,

Check out my update please:
http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg01318.html

Using aio=native,cache=none results in 500%-2000% performance drop comparing to bare metal and 300%-1000% comparing to aio=threads,cache=unsafe.

I'd like to know what numbers should I expect. Can somebody share what results do you have with aio=native in sequential IO ops? And of course please add some info about disk and controller configuration.
Maybe there's some bug in current version of Qemu in Debian 8 - testing. Package details: https://packages.debian.org/jessie/qemu-kvm . Version: qemu-kvm (1:2.1+dfsg-11)

On 04/13/2015 03:45 AM, Fam Zheng wrote:
On Fri, 04/10 22:38, ein wrote:

Qemu creates more than 70 threads and everyone of them tries to write to
disk, which results in:
1. High I/O time.
2. Large latency.
2. Poor sequential read/write speeds.

When I limited number of cores, I guess I limited number of threads as
well. That's why I got better numbers.

I've tried to combine AIO native and thread setting with deadline
scheduler. Native AIO was much more worse.

The final question, is there any way to prevent Qemu for making so large
number of processes when VM does only one sequential R/W operation?

aio=native will make QEMU only submit IO from the IO thread, so you shouldn't
see 70 threads with that. And that should usually be a better option for
performance.


Fam

--------------060200000403050204070502-- --------------060504050006020300040305 Content-Type: application/pgp-keys; name="0xF2C6EA10.asc" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="0xF2C6EA10.asc" -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v2.0.19 (GNU/Linux) mQINBFIvGgYBEADZ96zOyz3HuWYwaM4NzFnxJEl+ANCi2stqCdTJFZqRlJYHNVKC b5KlTI9ctkhfGEwNMsGEU4Ox68U8+JIp6jR4e+yyCs8E5t1djlj+WFqJcb7nenQ4 wqr1PW2dangJ3i6ORwuJlsx1rUlVY5rkYwFGwo30EezJ8KXSjMFt27JI8CT9PX16 jEmp2qxj/Z4LfkLDusQ6PbiRwb6TflAENSTH6bVzHK0cNH4YRZa1DjK9JX+4Bfkh 5rECDmSUxLjRV3xnMrpooQmSY5FB3zPl2zUMmrliC0TdFqVZ4BU5JBULq9WvggBY 5OvYy1BkQOu0ZiRuIjygNSmKs3kCSzUhgHg33R8QsGrP68Bev4m9uqYc2odFjr3W 26WF4xIJ5upQbbnnkLaZWoYTNmtuzVfQEcO38ELrE9yfKR8Xk15WeknoOX2cWuvh rRAPyqaUHnrkMWxAggvzfFuwXNIv5ctabWbo25P2lzIoAM3oOyajbSJW/4eiZyDB cR01LrKpur8jMyCR3AViJH3j5E5Bg/uYV2LlTMLb6t220uYq9U2IAK5t33bLyNhz 4rW7x8bZVvtak5MOldK5R3nAXuRdAgjV8ymWQABXQMZqmdm0Z1Q1l1GVm+i5wMRQ XND5k2+O655PXIAAwasmLQsfIqwRDzCh90JzFnfIY9dwjq6LgnpBMYxUowARAQAB tDlTb2tvxYJvd3NraSBNaWNoYcWCIChubyBwYXNzd29yZCkgPG1zb2tvbG93c2tp QGluYm94LmNvbT6JAj4EEwECACgFAlIvGgYCGyMFCQ0oaIAGCwkIBwMCBhUIAgkK CwQWAgMBAh4BAheAAAoJEHr/aeHyxuoQruwQAJh+zY6y9icu/heUrApLeKYr3am5 O71rI+TKnVnCOOS0n4XCkRDEW0UrXy3IOZIhIUZX0Fc5YHJPOfnfot6DvtgDvJnN K54Ik4k9rQNw2J+Iei4egDyTIDlYMOXwViHB9gFU3keuo/Pf2Jrn1bwVE4o66tQM 7E6V+hh20boQ1u95f3nb+81NX/38mb5a8QtCd1/B69tZc8HWwhViv9Of97fbtZ2R z2adnQT6mdYerMLxtag1TLq2ppOoVNbv6RR3X0qbeCm1VP6g3lAkGepwwPDADzd3 IIGnq+crvOPXsa6UkDJrgoyfAxlidneCampqdkepq1kfRPjRYkm5qZ1lrcm4NArf 1kSRI2VPQptTMeiqGQm58aIJOwoQpXe6lg7hZpOcF23JyCxRunpN3ov3tAjc+oVy 6umO5JZRr/LIqdjD7ogGqjZ4EGyAnlxD0wNSjkXCLb1ehqi+h0en19T3n5Xsoj54 nZJSjz1utwxLb8OLammrBBlNjdrbnjPhMT9pkiSJ7UMQjOtOe9rcSP3/dLu+durd Y8UZNheTcbqIb0DgTfn/HeZC93cf1+CJE8HwUrHCO712pWBuCTu+03JtmPKYFE9l BX6+LVfYhbfrjt64QXEJ/zN+OGtZoYI96yJeBbTQjaKk6DZXHroiRm0qnM68HUVK vMqpiG8wS/uSKUnEtCVlaW4ubmV0QGdtYWlsLmNvbSA8ZWluLm5ldEBnbWFpbC5j b20+iQI+BBMBAgAoBQJSLxseAhsjBQkNKGiABgsJCAcDAgYVCAIJCgsEFgIDAQIe AQIXgAAKCRB6/2nh8sbqEE6ED/93jZSp7d1aFa0UTDxsCmlveN1ti7+hQFpRVqRO duC3XEIClPZxXMI4mTNuHVQTUkIK2KHy1CLNKvyGOVZkaHELTV9+u5oYnZamqZ7E UtHbUFuIZThE2XLhz8/z5Q1575+rLUdus9J9GVYj2qI1decA2NJSWfrR9LWVygFj ICP70b3hEwLiQWUDYBUNXkA+BwM0BFz0mwy+ci8kMuo7pM0D9M2wQcgRj5VPFVKI Q3/tEthN3rZdwDebgvd1f3+FZ97HKR1GJt8n0EC0O5cIehso09rLnLdDfggxdYg3 oGhK50ZIy0oqUJiTUVTdyh2PrOG2pybR1XnHqIWObRsEWfnVJiIWoPKAkGegqfU+ S+M1MrJXlMiFaU16FIDdjK8Pai8bcm4S3GAasttOegG5MirBRTTgI7lqFkSMi4aj TvQvgwT4VgzoWcHrMXpQkyPyzOVESTW5nbn3jLuU9HIl55UKzi0HTmUd0z/pwWAO Djsq8JQO1IbU5/iNTad4F2HfPWp21zvvhzYH6Ihy3Fzcjv2cjfPfZFxJmfmEqsIK Vvvj+YbHEHBi0dkWF3Dkf56e/eA96DcAUyMvAwvp38cl1K6mEHRLfn3YmoW1dqwL p7GrvrnhS+Jb+6px/atvw+fVjk7OfQY1wkcsu5oxFUY7NxkXwHpNi7aIM8tNGiwu LMcSILkCDQRSLxoGARAAr0cXmo0WBLsQj1JAW3sMd+bzCIfv2tOHbK2Rh/NvwwF2 LaLHmmLaEuFzMihdBLwZdgXTVi7EOdLcHK7qlTxYaNAOJYh2e3oBhSi3CBbvVg/S vrIvWKjAuMr7zuJWwFMWZsXp9iuaGf3OKHPkF4R2kxWC3JWcE/DTiejxu0QpLc3y Un7J3zAAep/LAmYbKg+YXH72TaVm6pXcktW1CUGcu32T1XtupJNFZRvT2OBHXyhw Ay+N7tfWGF+wJuOAYLfhdM/MWsrWirHk8mmRYSZtJmCcpmQe5kBHgYhDyqj5nS3q mg1HIJCnupPMe6tdD0836eGVrimTj4aOJE2UAPdgwBvW5j7Wx/vrS1L4qYdiCwK6 KK95qNv5GwfSj5+3unq4FDLWtAeuAdQeCNGhs4RWyOxwHFxaFkJAspGYsRu5ue3w +ftELeVY/e/IaQFykKIYzGBKrv0ASyQOsd0U3I8BNWmhkKlFz8l+Sdw3Co+mvGM6 M/SHQTAefe6/aaxbyEbEjA6s//i1DeBPFtLSxzR9s/Vkv+GJZzhIyYdQShZ9TPgv tSeVS1KVlLWDRK+qrixFIr/3oeZGv8sz3FU9xa7eODCRcFcUQO+mG2fBDztF5i4C HyQIa/M65c8GQxpe/IhMFVyFWDrLLFozLyT1NgYWShcag14+bpReZkRBdnknF8UA EQEAAYkCJQQYAQIADwUCUi8aBgIbDAUJDShogAAKCRB6/2nh8sbqEHHCEACvwgON ZsG7jywBhYFKk90AoZgDjQ47utRREvKshi09R2pMUGrfGwhMhJvFsr65aEY2Q6Sx /SKBb06uBlAyGoedBNwSy8uN+0Iv5sDMnz78KpS9MG/eF4e/5KT8zRn9JTdGQiTx VMAK3rMKbcYrp6VpzYyAbZpR93MvFXauFJFc6fy8hFmhCDDtxtwkVwGqpl+pCk7x pAERp6T0/TXJpGKuFNgZ9oONFbBWOaDnqEP+NHttNPAz40nDb4I85g65zDe26MQv GVmmYHHmNzesOqwTAHxT6KfEwo15+cv3S2GY52WG9PpYXsbapOmN4R41i0/hu/NE RUNVk1YOR7vvpNMgRQ54oXv04aCDNwdOPeCbhSwqfddbYpGUz4yGe6lL677YMJoB JnMsse5Em6jV2av73QUFel+s4DlWeko2b3ZERLEWm04EclC8pAWmkewdPNhOJ0Va W5pfG6a0/NKIQCXxK3wkkXQpmKMdemyE7j43Hy0C3/40IkvQKbwlygad6NiCz862 a53Z/oGuvYcQbt5rye54xhEGgoc0KmPmvretcoKm2udFrDkSjoBfS5FM40Pc6/4L abxySE+B/h2JjTOJUjsxmsMwWexFd9heUrgF30HGW7B5GFMR0edMsTfyn1HN1MYZ 3cTLfZCrJDxCJ022/Kajr+QOjjaNI7s4uxpizg=3D=3D =3Dw/Gc -----END PGP PUBLIC KEY BLOCK----- --------------060504050006020300040305--