From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53335) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c48Ul-0005Fj-3q for qemu-devel@nongnu.org; Tue, 08 Nov 2016 10:39:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c48Ug-0008G4-63 for qemu-devel@nongnu.org; Tue, 08 Nov 2016 10:39:31 -0500 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]:36683) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1c48Uf-0008Fl-Uz for qemu-devel@nongnu.org; Tue, 08 Nov 2016 10:39:26 -0500 Received: by mail-wm0-x244.google.com with SMTP id c17so23339678wmc.3 for ; Tue, 08 Nov 2016 07:39:25 -0800 (PST) Date: Tue, 8 Nov 2016 15:39:15 +0000 From: Stefan Hajnoczi Message-ID: <20161108153915.GA11274@stefanha-x1.localdomain> References: <1475035789-685-1-git-send-email-ashish.mittal@veritas.com> <20160930083606.GA5411@stefanha-x1.localdomain> <20161104095223.GC9817@stefanha-x1.localdomain> <87C0A0E2-9D12-4C54-81F9-07C5200A5306@veritas.com> <20161107102202.GA5036@stefanha-x1.localdomain> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YZ5djTAD1cGYuMQK" Content-Disposition: inline In-Reply-To: Subject: Re: [Qemu-devel] [PATCH v7 RFC] block/vxhs: Initial commit to add Veritas HyperScale VxHS block device support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Ketan Nilangekar Cc: Ashish Mittal , "qemu-devel@nongnu.org" , "pbonzini@redhat.com" , "kwolf@redhat.com" , "armbru@redhat.com" , "berrange@redhat.com" , "jcody@redhat.com" , "famz@redhat.com" , Ashish Mittal , Abhijit Dey --YZ5djTAD1cGYuMQK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 07, 2016 at 08:27:39PM +0000, Ketan Nilangekar wrote: > On 11/7/16, 2:22 AM, "Stefan Hajnoczi" wrote: > >On Fri, Nov 04, 2016 at 06:30:47PM +0000, Ketan Nilangekar wrote: > >> > On Nov 4, 2016, at 2:52 AM, Stefan Hajnoczi wro= te: > >> >> On Thu, Oct 20, 2016 at 01:31:15AM +0000, Ketan Nilangekar wrote: > >> >> 2. The idea of having multi-threaded epoll based network client was= to drive more throughput by using multiplexed epoll implementation and (fa= irly) distributing IOs from several vdisks (typical VM assumed to have atle= ast 2) across 8 connections.=20 > >> >> Each connection is serviced by single epoll and does not share its = context with other connections/epoll. All memory pools/queues are in the co= ntext of a connection/epoll. > >> >> The qemu thread enqueues IO request in one of the 8 epoll queues us= ing a round-robin. Responses are also handled in the context of an epoll lo= op and do not share context with other epolls. Any synchronization code tha= t you see today in the driver callback is code that handles the split IOs w= hich we plan to address by a) implementing readv in libqnio and b) removing= the 4MB limit on write IO size. > >> >> The number of client epoll threads (8) is a #define in qnio and can= easily be changed. However our tests indicate that we are able to drive a = good number of IOs using 8 threads/epolls. > >> >> I am sure there are ways to simplify the library implementation, bu= t for now the performance of the epoll threads is more than satisfactory. > >> >=20 > >> > By the way, when you benchmark with 8 epoll threads, are there any o= ther > >> > guests with vxhs running on the machine? > >> >=20 > >>=20 > >> Yes. Infact the total througput with around 4-5 VMs scales well to sat= urate around 90% of available storage throughput of a typical pcie ssd devi= ce. > >>=20 > >> > In a real-life situation where multiple VMs are running on a single = host > >> > it may turn out that giving each VM 8 epoll threads doesn't help at = all > >> > because the host CPUs are busy with other tasks. > >>=20 > >> The exact number of epolls required to achieve optimal throughput may = be something that can be adjusted dynamically by the qnio library in subseq= uent revisions.=20 > >>=20 > >> But as I mentioned today we can change this by simply rebuilding qnio = with a different value for the #define > > > >In QEMU there is currently work to add multiqueue support to the block > >layer. This enables true multiqueue from the guest down to the storage > >backend. >=20 > Is there any spec or documentation on this that you can point us to? The current status is: 1. virtio-blk and virtio-scsi support multiple queues but these queues are processed from a single thread today. 2. MemoryRegions can be marked with !global_locking so its handler functions are dispatched without taking the QEMU global mutex. This allows device emulation to run in multiple threads. 3. Paolo Bonzini (CCed) is currently working on make the block layer (BlockDriverState and co) support access from multiple threads and multiqueue. This is work in progress. If you are interested in this work keep an eye out for patch series from Paolo Bonzini and Fam Zheng. Stefan --YZ5djTAD1cGYuMQK Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJYIfGjAAoJEJykq7OBq3PI3O8IAJJ3lJONxDGSZss7pGyc/B4a yksmA7gnItimkDsFDGLSItF/L7tf/vmW6ubMTiJ9334+5YQ0mg9i6eqxpOHfB1nH MB6L3sYd2XKXSz/39rQ31Iadn2dTAyhmXT30L/Azg80FerQzOqL24Njz+nJAH/ZI +Jkx6VqW/QiKO70lRA4DiINTV+X5A2s1X+4sWQi48gXVV6GLP27xEXz4owdLXM8K vmkFliumN28aWD8bzlV4sGq3BVe6ezkaiAW2BYNyB0W7IOh58n7aDokm2/Wr36Pm F23jBd07CpSCnefTyp9Wnj0WUR6XPYNjoHYSl2dT7O0m9/0NfKr12tHyBA2EDR0= =pa+J -----END PGP SIGNATURE----- --YZ5djTAD1cGYuMQK--