From mboxrd@z Thu Jan 1 00:00:00 1970 From: Willem de Bruijn Subject: Re: [PATCH RFC net-next 5/6] rds: support for zcopy completion notification Date: Thu, 18 Jan 2018 17:46:26 -0500 Message-ID: References: <9ff41bc8f61a112138287b5029369a9910477811.1516147540.git.sowmini.varadhan@oracle.com> <20180118114024.GB24920@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Cc: Network Development , David Miller , rds-devel@oss.oracle.com, santosh.shilimkar@oracle.com To: Sowmini Varadhan Return-path: Received: from mail-oi0-f50.google.com ([209.85.218.50]:35520 "EHLO mail-oi0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932510AbeARWrI (ORCPT ); Thu, 18 Jan 2018 17:47:08 -0500 Received: by mail-oi0-f50.google.com with SMTP id b11so4144087oif.2 for ; Thu, 18 Jan 2018 14:47:08 -0800 (PST) In-Reply-To: <20180118114024.GB24920@oracle.com> Sender: netdev-owner@vger.kernel.org List-ID: >> This changes the semantics of these fields. Please add a new SO_EE_CODE flag, >> even if the semantics can be derived from the packet family (for now). > > sounds good, I'll take care of this (and other review comments) > for the next rev. > >> Even better would be if we can avoid the cookies completely. I understand >> the issue with concurrent send threads racing on obtaining a zckey value. If >> the sender could learn which zckey was chosen for a call, would that suffice? > > I'm not sure I understand the proposal- you want sendmsg to return > the cookie ("zckey") for you? How? > > even if we mangled sendmsg() to return something other than > the existing POSIX semantics, how will the application be asynchronously > notified that send has completed (on a per-message/per-datagram) basis? I'm purposely glossing over how the kernel returns this item for now. Was just wondering whether we can then assume mostly in order delivery and reuse the existing notification interface from tcp zerocopy. >>From your experiments, it sounds like this is not the case. In which case there is little benefit to trying to force linear IDs derived from sk->sk_zckey. >> I suspect that in even with concurrent senders, notifications arrive >> largely in >> order, in which case we could just maintain the existing semantics and even >> reuse that implementation. > > not so. rds-stress [1] with -d8 -t8 quickly disproves this on my 10G ixgbe > connection. Okay. In that case, the cmsg cookie approach sounds fine. I had the same in an early tcp zerocopy test version, as a matter of fact. >> > } rdma; >> > struct rm_data_op { >> > unsigned int op_active:1; >> > - unsigned int op_notify:1; >> > + unsigned int op_notify:1, >> > + op_zcopy:1, > : >> > + struct rds_znotifier *op_mmp_znotifier; >> >> not necessary if op_mmp_znotifier is NULL unless set in >> rds_message_copy_from_user > > To make sure I dont misunderstand, you are suggesting that we dont > need op_zcopy, but can just check for the null-ness of > op_mmp_znotifier (yes, true, I agree)? or something else? Yes, that's what I meant.