From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AF4BC2D0C2 for ; Fri, 3 Jan 2020 12:39:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 02DAD20848 for ; Fri, 3 Jan 2020 12:39:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=cloud.ionos.com header.i=@cloud.ionos.com header.b="ehY6ksng" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727521AbgACMjk (ORCPT ); Fri, 3 Jan 2020 07:39:40 -0500 Received: from mail-io1-f66.google.com ([209.85.166.66]:43879 "EHLO mail-io1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727507AbgACMjk (ORCPT ); Fri, 3 Jan 2020 07:39:40 -0500 Received: by mail-io1-f66.google.com with SMTP id n21so39636484ioo.10 for ; Fri, 03 Jan 2020 04:39:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloud.ionos.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=cAe5HcMa95ZS+hogq0Ch8E/KQ7+bOnNlPN6CHTvuwVc=; b=ehY6ksngLD2Alabqsy7PAjXpgPvGNIqTto5LLJyk/an47FZX1K+1cMUNrvDBbCWmZG CoG0bAeR13drjgQ8PU8Bk2Ml2ZgQqNVVofB+88+ORb4GtqRJ5og/TphzQknZ0ZkDqShA BuewhbiTHIOZ+CUNlGyDWUGCLwPjC0tTGuJTeQ3ayrO0oMoET+/dyP3wyLjk+XzBSn2Q IMpst+i2G3qZ6BKg7Z0H16Itds45wjtYppREzXf9TkPrc55tmE8fdNzY8NyLFmhemxK+ nwM+9ijpkLwwCh+WBHBl6pLBmQgQ9RaDhzUMsQJNLlk5WZpypa1nWcxinkp+nhBzlB11 SuCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cAe5HcMa95ZS+hogq0Ch8E/KQ7+bOnNlPN6CHTvuwVc=; b=Ixz+a6e3T51e+DK21G12pAQdtOyBul0DrMOmJTFSmNKss4cTdXYCpwA20BwhcOEGPh KPDscGdj001AcQ05IMk1LB+MVkEylQ2hIRpmtgileVwWUdp90BohFaBejdsR8wI7GdAZ HBTpqvEQMgeSmcXBCld9IF+KgPP1+niyNtPJ+BOVs9akVuup77tf8FlQzRaAjBY2vral jsd21rIMECxrFfejhzAlpth0UC6ZDmDQycHgrOmJd6XPZNkMfL75hWvCEOAmHsmANdmU yvlD/4V/Q5sBW4LDDHDHGS19SaMhBpA52SUZdnh0ISyEH+K/JnRcN8EkgxAIxCBfwrgo RSWw== X-Gm-Message-State: APjAAAXOddNDGPfHIor/FdCo8Yi08t3j30ScnQYcVhZ2wAWUttqJkB5S vBfpcRXM7pXKi4HM+cjCD1N3iXyx/WjjzVt9aSA4dg== X-Google-Smtp-Source: APXvYqw9gUHKrjfDdiAi42fotiG2xzcc+wApMPhqxzYw3KcG/z9y36HbKX9/ogOj21lxFmI9dCVtjZ63vbDTrvpk8YI= X-Received: by 2002:a5d:84d6:: with SMTP id z22mr55730403ior.54.1578055179696; Fri, 03 Jan 2020 04:39:39 -0800 (PST) MIME-Version: 1.0 References: <20191220155109.8959-1-jinpuwang@gmail.com> <20200102181859.GC9282@ziepe.ca> In-Reply-To: <20200102181859.GC9282@ziepe.ca> From: Jinpu Wang Date: Fri, 3 Jan 2020 13:39:29 +0100 Message-ID: Subject: Re: [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device To: Jason Gunthorpe Cc: Jack Wang , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, Jens Axboe , Christoph Hellwig , Sagi Grimberg , Bart Van Assche , Leon Romanovsky , Doug Ledford , Danil Kipnis , rpenyaev@suse.de Content-Type: text/plain; charset="UTF-8" Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Thu, Jan 2, 2020 at 7:19 PM Jason Gunthorpe wrote: > > On Fri, Dec 20, 2019 at 04:50:44PM +0100, Jack Wang wrote: > > Hi all, > > > > here is V5 of the RTRS (former IBTRS) rdma transport library and the > > corresponding RNBD (former IBNBD) rdma network block device. > > > > Main changes are the following: > > 1. Fix the security problem pointed out by Jason > > 2. Implement code-style/readability/API/etc suggestions by Bart van Assche > > 3. Rename IBTRS and IBNBD to RTRS and RNBD accordingly > > 4. Fileio mode support in rnbd-srv has been removed. > > > > The main functional change is a fix for the security problem pointed out by > > Jason and discussed both on the mailing list and during the last LPC RDMA MC 2019. > > On the server side we now invalidate in RTRS each rdma buffer before we hand it > > over to RNBD server and in turn to the block layer. A new rkey is generated and > > registered for the buffer after it returns back from the block layer and RNBD > > server. The new rkey is sent back to the client along with the IO result. > > The procedure is the default behaviour of the driver. This invalidation and > > registration on each IO causes performance drop of up to 20%. A user of the > > driver may choose to load the modules with this mechanism switched > > off > > So, how does this compare now to nvme over fabrics? > > I recall there were questiosn why we needed yet another RDMA block > transport? > > Jason Performance results for the v5.5-rc1 kernel are here: link: https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v5-v5.5-rc1 Some workloads RNBD are faster, some workloads NVMeoF are faster.