From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F4A9C2D0C2 for ; Thu, 2 Jan 2020 18:23:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 34D0121582 for ; Thu, 2 Jan 2020 18:23:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="BbBQ8pHx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727983AbgABSX6 (ORCPT ); Thu, 2 Jan 2020 13:23:58 -0500 Received: from mail-qt1-f195.google.com ([209.85.160.195]:37897 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727976AbgABSX6 (ORCPT ); Thu, 2 Jan 2020 13:23:58 -0500 Received: by mail-qt1-f195.google.com with SMTP id n15so35262864qtp.5 for ; Thu, 02 Jan 2020 10:23:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=zAwqsRR5p0WQ1FXAD94ssaK9uFiL3MMXtJkj40Y5ME4=; b=BbBQ8pHxMSCg1FRK2/9Xmge46DvZ5pwR6dQgQSqPyKWaSX7skW6fo+eapjhmTaGZVe HgbdRwsGddZnRBTZ9ol07RyiJFe5oQTm10XNPdQFLcMycphSPZIBF7ZIZlMQQPNZjH+W PiXPnRynCswGNrk5s8mVsQnmeJE7XTsuIJkpdnVjSFDqi61p/OpKvyz6ATc0emGr4UVZ z3IrEIdfXrXOGeLAoihJL0Lc5U+zNjxhnVmR2NO9OyFcTs5XUBWfnFdobWzh9chWY0L5 vjJ3oW9Pn2lXXjNYjslpGXMCrFKWMjQwZhiqWLelNpwQkQNZsG56EtKWyI9l+XGKL7mr 0HYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=zAwqsRR5p0WQ1FXAD94ssaK9uFiL3MMXtJkj40Y5ME4=; b=QZjncJ2sLrLbBPOqbT4LJusBziH0r21aumltkNXzf5OozpJC5ggkEXB1Ytb3YszRv6 kPgiXuA+vbTjaWpoHjmoavzuBmY44al3ebkMuR28OJkUHFOp0YIw65pyONzUc1f9pN2b 5iaLb2aqsP7d4hYfkSXqt7WqjXIQ7r+zlFvEzrm/nfBZde6sSwIIkwX2vzW6dbXBBvZW DxnpIajpvyyI1nMnf8vPgJ4TYir1QJz2741fE3hx2nDUvzieyxmDQmTa3nCnXyR+sfWQ pszZbkxMbra4HMPFeGuFG/QNiP6rzoseaG20oXQM8BZ04G505XSAezOCBkAC6bgs9v4I HFmA== X-Gm-Message-State: APjAAAVAfDz518CRLoUMbupp4yaqenCTg9CawHsVy7HNZ7Au0NJNq0BN IGpe3dvPNp/CIfZ/3W6QvSU+wQ== X-Google-Smtp-Source: APXvYqzlkWLLP2XjfL2M0f5z2qad2eqD1MLtK2QP0ABvzi+Up6KAgHVirMSsFqRfcngstb+4Qbunjw== X-Received: by 2002:ac8:3703:: with SMTP id o3mr60602438qtb.208.1577989437359; Thu, 02 Jan 2020 10:23:57 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id i6sm15312719qkk.7.2020.01.02.10.23.56 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 02 Jan 2020 10:23:56 -0800 (PST) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1in58W-0003Pg-HN; Thu, 02 Jan 2020 14:23:56 -0400 Date: Thu, 2 Jan 2020 14:23:56 -0400 From: Jason Gunthorpe To: Bart Van Assche Cc: Jack Wang , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, axboe@kernel.dk, hch@infradead.org, sagi@grimberg.me, leon@kernel.org, dledford@redhat.com, danil.kipnis@cloud.ionos.com, jinpu.wang@cloud.ionos.com, rpenyaev@suse.de Subject: Re: [PATCH v6 06/25] rtrs: client: main functionality Message-ID: <20200102182356.GD9282@ziepe.ca> References: <20191230102942.18395-1-jinpuwang@gmail.com> <20191230102942.18395-7-jinpuwang@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Mon, Dec 30, 2019 at 03:53:10PM -0800, Bart Van Assche wrote: > > + if (req->sg_cnt) { > > + if (unlikely(req->dir == DMA_FROM_DEVICE && req->need_inv)) { > > + /* > > + * We are here to invalidate RDMA read requests > > + * ourselves. In normal scenario server should > > + * send INV for all requested RDMA reads, but > > + * we are here, thus two things could happen: > > + * > > + * 1. this is failover, when errno != 0 > > + * and can_wait == 1, > > + * > > + * 2. something totally bad happened and > > + * server forgot to send INV, so we > > + * should do that ourselves. > > + */ > > Please document in the protocol documentation when RDMA reads are used. > > What does "server forgot to send INV" mean? > > Additionally, if I remember correctly Jason considers it very important > that invalidation happens from the submitting context because otherwise > the RDMA retry mechanism can't work. I think my point has usually been you can't use completions on the RQ to deduce the state of the SQ But if you are doing inv by posting it on the same SQ then things will get ordered OK as the HW shouldn't progress the INV until any work touching that rkey is also concluded. Jason