From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754823AbbBOIUJ (ORCPT ); Sun, 15 Feb 2015 03:20:09 -0500 Received: from userp1040.oracle.com ([156.151.31.81]:19909 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754480AbbBOIUF (ORCPT ); Sun, 15 Feb 2015 03:20:05 -0500 From: Bob Liu To: xen-devel@lists.xen.org Cc: david.vrabel@citrix.com, linux-kernel@vger.kernel.org, roger.pau@citrix.com, konrad.wilk@oracle.com, felipe.franciosi@citrix.com, axboe@fb.com, hch@infradead.org, avanzini.arianna@gmail.com, Bob Liu Subject: [RFC PATCH 00/10] Multi-queue support for xen-block driver Date: Sun, 15 Feb 2015 16:18:55 +0800 Message-Id: <1423988345-4005-1-git-send-email-bob.liu@oracle.com> X-Mailer: git-send-email 1.7.10.4 X-Source-IP: ucsinet22.oracle.com [156.151.31.94] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patchset convert the Xen PV block driver to the multi-queue block layer API by sharing and using multiple I/O rings between the frontend and backend. History: It's based on the result of Arianna's internship for GNOME's Outreach Program for Women, in which she was mentored by Konrad Rzeszutek Wilk. I also worked on this patchset with her at that time, and now fully take over this task. I've got her authorization to "change authorship or SoB to the patches as you like." A few words on block multi-queue layer: Multi-queue block layer improved block scalability a lot by split single request queue to per-processor software queues and hardware dispatch queues. The linux blk-mq API will handle software queues, while specific block driver must deal with hardware queues. The xen/block implementation: 1) Convert to blk-mq api with only one hardware queue. 2) Use more rings to act as multi hardware queues. 3) Negotiate number of hardware queues, the same as xen-net driver. The backend notify "multi-queue-max-queues" to frontend, then the front write back final number to "multi-queue-num-queues". Test result: fio's IOmeter emulation on a 16 cpus domU with a null_blk device, hardware queue number was 16. nr_fio_jobs IOPS(before) IOPS(after) Diff 1 57k 58k 0% 4 95k 201k +210% 8 89k 372k +410% 16 68k 284k +410% 32 65k 196k +300% 64 63k 183k +290% More results are coming, there was also big improvement on both write-IOPS and latency. Any comments or suggestions are welcome. Thank you, -Bob Liu Bob Liu (10): xen/blkfront: convert to blk-mq API xen/blkfront: drop legacy block layer support xen/blkfront: reorg info->io_lock after using blk-mq API xen/blkfront: separate ring information to an new struct xen/blkback: separate ring information out of struct xen_blkif xen/blkfront: pseudo support for multi hardware queues xen/blkback: pseudo support for multi hardware queues xen/blkfront: negotiate hardware queue number with backend xen/blkback: get hardware queue number from blkfront xen/blkfront: use work queue to fast blkif interrupt return drivers/block/xen-blkback/blkback.c | 370 ++++++++------- drivers/block/xen-blkback/common.h | 54 ++- drivers/block/xen-blkback/xenbus.c | 415 +++++++++++------ drivers/block/xen-blkfront.c | 894 +++++++++++++++++++++--------------- 4 files changed, 1018 insertions(+), 715 deletions(-) -- 1.8.3.1 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bob Liu Subject: [RFC PATCH 00/10] Multi-queue support for xen-block driver Date: Sun, 15 Feb 2015 16:18:55 +0800 Message-ID: <1423988345-4005-1-git-send-email-bob.liu@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org Cc: hch@infradead.org, felipe.franciosi@citrix.com, linux-kernel@vger.kernel.org, axboe@fb.com, Bob Liu , david.vrabel@citrix.com, avanzini.arianna@gmail.com, roger.pau@citrix.com List-Id: xen-devel@lists.xenproject.org This patchset convert the Xen PV block driver to the multi-queue block layer API by sharing and using multiple I/O rings between the frontend and backend. History: It's based on the result of Arianna's internship for GNOME's Outreach Program for Women, in which she was mentored by Konrad Rzeszutek Wilk. I also worked on this patchset with her at that time, and now fully take over this task. I've got her authorization to "change authorship or SoB to the patches as you like." A few words on block multi-queue layer: Multi-queue block layer improved block scalability a lot by split single request queue to per-processor software queues and hardware dispatch queues. The linux blk-mq API will handle software queues, while specific block driver must deal with hardware queues. The xen/block implementation: 1) Convert to blk-mq api with only one hardware queue. 2) Use more rings to act as multi hardware queues. 3) Negotiate number of hardware queues, the same as xen-net driver. The backend notify "multi-queue-max-queues" to frontend, then the front write back final number to "multi-queue-num-queues". Test result: fio's IOmeter emulation on a 16 cpus domU with a null_blk device, hardware queue number was 16. nr_fio_jobs IOPS(before) IOPS(after) Diff 1 57k 58k 0% 4 95k 201k +210% 8 89k 372k +410% 16 68k 284k +410% 32 65k 196k +300% 64 63k 183k +290% More results are coming, there was also big improvement on both write-IOPS and latency. Any comments or suggestions are welcome. Thank you, -Bob Liu Bob Liu (10): xen/blkfront: convert to blk-mq API xen/blkfront: drop legacy block layer support xen/blkfront: reorg info->io_lock after using blk-mq API xen/blkfront: separate ring information to an new struct xen/blkback: separate ring information out of struct xen_blkif xen/blkfront: pseudo support for multi hardware queues xen/blkback: pseudo support for multi hardware queues xen/blkfront: negotiate hardware queue number with backend xen/blkback: get hardware queue number from blkfront xen/blkfront: use work queue to fast blkif interrupt return drivers/block/xen-blkback/blkback.c | 370 ++++++++------- drivers/block/xen-blkback/common.h | 54 ++- drivers/block/xen-blkback/xenbus.c | 415 +++++++++++------ drivers/block/xen-blkfront.c | 894 +++++++++++++++++++++--------------- 4 files changed, 1018 insertions(+), 715 deletions(-) -- 1.8.3.1