From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF93CC4360C for ; Fri, 27 Sep 2019 12:18:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A3D4C217D9 for ; Fri, 27 Sep 2019 12:18:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=cloud.ionos.com header.i=@cloud.ionos.com header.b="HdrofhHQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727393AbfI0MSJ (ORCPT ); Fri, 27 Sep 2019 08:18:09 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:39921 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725890AbfI0MSI (ORCPT ); Fri, 27 Sep 2019 08:18:08 -0400 Received: by mail-ed1-f66.google.com with SMTP id a15so2138827edt.6 for ; Fri, 27 Sep 2019 05:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloud.ionos.com; s=google; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=FKHeGQHPDqERmEIFxtVPQVkrwntjuPpxcT3RmSIpP3o=; b=HdrofhHQzBPK9v8Uo6SaER6UTr3ErMkOtpIMUvNAfRk3TsOnPbuZUuyzDLpriySmte SoF9tI5+/jTLLvfjl+U399goUWoufGPSSthoOvnAfh/8OD8i9WBqMC6tZsLN6LSvE5Zk 9m3+kreK+0+6KaLuL35SO7qZNvzuC0iexA2pE/RHotailaByQjeWaqyVI8OD2J2MYj+1 rik304UZpq+4QIF74Y4WXzaduvZaZUo5rxadnZY7tWJUYxnyHSvQd7k5ujOLIzXGCWcU bJb4nF0aGfEDv+rasjeCEjqiEJ3Rfsef0zm8uhvdWfXKw/5JmM7WoRTMJpfrpsHDOjk9 8n+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=FKHeGQHPDqERmEIFxtVPQVkrwntjuPpxcT3RmSIpP3o=; b=WY5r7D3PvpjOXXKxVXTlTaYnrEjxruBx9oyKlSQ8jDdMVon/x4XjmwiaDvsBujWc/O LAXRN4NINqYApzlH4QJnKgJUQ4yWrJC2/jhVmUbQwl/t4d7a1N+ARm2k1so3bWb95MLx jswe3W81wbPmOTk+W9JV7n0oc7NBhfk569soA/eCY2Zy0JW042DF7WGYItdx6Tsrhjsg kyKz4Llt5eldZ0Nzph3wvZ7zIYKiwWEyseG+98pMrR3fJGOO6Hs6mZ/vQsXVRE6MLquO wKa3ptXRiXISryYZdkafLweQnvjl2LFLpcdN5FxFfpq4p4IQFm9WZKT9appJB7Da9MKx 3jhQ== X-Gm-Message-State: APjAAAUqw0ZsKAE+UCEpnQusOgK/qoUlmIztPU5zqi/0C9QibSPtJjUc lKGMSkYoxSftSjV9jD4tavs+ X-Google-Smtp-Source: APXvYqy1JTYW4Su7MKrX0sOFsS3sjuB529ktql8CtIDV3FED1QGblp/9SVcsm2Y+sy5ezO3svyfbFw== X-Received: by 2002:a50:91b1:: with SMTP id g46mr3998224eda.255.1569586686552; Fri, 27 Sep 2019 05:18:06 -0700 (PDT) Received: from ?IPv6:2a02:247f:ffff:2540:251d:77f4:c17c:46f7? ([2001:1438:4010:2540:251d:77f4:c17c:46f7]) by smtp.gmail.com with ESMTPSA id i7sm492490edk.42.2019.09.27.05.18.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Sep 2019 05:18:05 -0700 (PDT) Subject: Re: [PATCH v4 17/25] ibnbd: client: main functionality From: Danil Kipnis To: Roman Penyaev , Christoph Hellwig Cc: Bart Van Assche , Jack Wang , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, Jens Axboe , Sagi Grimberg , Jason Gunthorpe , Doug Ledford , rpenyaev@suse.de, Jack Wang References: <20190620150337.7847-1-jinpuwang@gmail.com> <20190620150337.7847-18-jinpuwang@gmail.com> <5c5ff7df-2cce-ec26-7893-55911e4d8595@acm.org> <6f677d56-82b3-a321-f338-cbf8ff4e83eb@acm.org> Message-ID: Date: Fri, 27 Sep 2019 14:18:05 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On 27.09.19 11:32, Danil Kipnis wrote: > On Fri, Sep 27, 2019 at 10:52 AM Roman Penyaev wrote: >> No, it seems this thingy is a bit different. According to my >> understanding patches 3 and 4 from this patchset do the >> following: 1# split equally the whole queue depth on number >> of hardware queues and 2# return tag number which is unique >> host-wide (more or less similar to unique_tag, right?). >> >> 2# is not needed for ibtrs, and 1# can be easy done by dividing >> queue_depth on number of hw queues on tag set allocation, e.g. >> something like the following: >> >> ... >> tags->nr_hw_queues = num_online_cpus(); >> tags->queue_depth = sess->queue_deph / tags->nr_hw_queues; >> >> blk_mq_alloc_tag_set(tags); >> >> >> And this trick won't work out for the performance. ibtrs client >> has a single resource: set of buffer chunks received from a >> server side. And these buffers should be dynamically distributed >> between IO producers according to the load. Having a hard split >> of the whole queue depth between hw queues we can forget about a >> dynamic load distribution, here is an example: >> >> - say server shares 1024 buffer chunks for a session (do not >> remember what is the actual number). >> >> - 1024 buffers are equally divided between hw queues, let's >> say 64 (number of cpus), so each queue is 16 requests depth. >> >> - only several CPUs produce IO, and instead of occupying the >> whole "bandwidth" of a session, i.e. 1024 buffer chunks, >> we limit ourselves to a small queue depth of an each hw >> queue. >> >> And performance drops significantly when number of IO producers >> is smaller than number of hw queues (CPUs), and it can be easily >> tested and proved. >> >> So for this particular ibtrs case tags should be globally shared, >> and seems (unfortunately) there is no any other similar requirements >> for other block devices. > I don't see any difference between what you describe here and 100 dm > volumes sitting on top of a single NVME device. Hallo Christoph, am I wrong? Thank you, Danil.