From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4491FC74A5B for ; Thu, 30 Mar 2023 01:46:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229791AbjC3Bqe (ORCPT ); Wed, 29 Mar 2023 21:46:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229672AbjC3Bqd (ORCPT ); Wed, 29 Mar 2023 21:46:33 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A61E4EEB for ; Wed, 29 Mar 2023 18:45:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680140745; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kv0JdKs3b/J2KtWtqtGbcl417IkaIZ3v3L6FuRPDYCY=; b=h/Uqi4yGR08up02Zp61RPKEXUyJbWh7djpHoEcS1vpEhk5cFUREN3od4Xp+SOdecmDEp++ uRgLPTR3Yt7sFw0/BDaNmO/dn5DO0A+6D1F3gB53VMxX1BYQ7LoJ6B7eRRcmnLeH68Ig62 AKn6DIFn+JwzjwWji6sVZEp/xxaVCdA= Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-237-X7vSrBHRNJ2t0ptqejN04w-1; Wed, 29 Mar 2023 21:45:44 -0400 X-MC-Unique: X7vSrBHRNJ2t0ptqejN04w-1 Received: by mail-pg1-f198.google.com with SMTP id q30-20020a631f5e000000b0050760997f4dso4793057pgm.6 for ; Wed, 29 Mar 2023 18:45:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680140743; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kv0JdKs3b/J2KtWtqtGbcl417IkaIZ3v3L6FuRPDYCY=; b=OCxbeF85g6C3UjvtpqZaFkAeOSKr3bwwvvacJKtxlCD3cEMf56K0XYwIUrG1z67/7d IrLa5RC/WHjtXcXap2RNpxBbt5WMc4CqJPBSZvILcACCeBR7pRM2jYke/FzJPuTyDyag YxHx9HfBu3k9+TSbRp/ZcwI8PyceRVIsD99duXQp8k7eR7q+wN30OzlHX2ni3R99JYNs SQ5srwNfmmRnNFF9egsRFYa++BYPhho2q9cKfraXK/c2QUR+r6pDHV+aPNgTfGffe8OF B9GvKKlrlz/srHmasv6t4jyM+bzAuitPwFYIAXpKoyKY/qagMrWT5zTX7A4U/q3qinW6 0Bzg== X-Gm-Message-State: AAQBX9fq0nIntby+4JuW8/bhfgXiyueJZaUTBowcjcrbYBeTaZII+WsN ddV+RDrJdnX+VI3AXvemfzmuNw15KVdAgNIqlwMIdfgDmxYkB6oaMkWXWrWSgR6J5gPiq860Jkt 33xuyvuKJogHy1BKLnCIGlwT43w== X-Received: by 2002:a17:903:22d2:b0:19a:7217:32a9 with SMTP id y18-20020a17090322d200b0019a721732a9mr4636118plg.26.1680140743029; Wed, 29 Mar 2023 18:45:43 -0700 (PDT) X-Google-Smtp-Source: AKy350ZC7R/sqsXLmXMgRh5yYmDYt07re8Mz0pqFOB7hKr/eVqrBonJ1fWeZySGpK6QRntg/4jjpKg== X-Received: by 2002:a17:903:22d2:b0:19a:7217:32a9 with SMTP id y18-20020a17090322d200b0019a721732a9mr4636092plg.26.1680140742679; Wed, 29 Mar 2023 18:45:42 -0700 (PDT) Received: from [10.72.12.51] ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id jj2-20020a170903048200b0019fea4bb887sm23709042plb.157.2023.03.29.18.45.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Mar 2023 18:45:42 -0700 (PDT) Message-ID: <7f7947d6-2a03-688b-dc5e-3887553f0106@redhat.com> Date: Thu, 30 Mar 2023 09:45:32 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: [RFC PATCH v2 37/48] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Content-Language: en-US To: David Howells , Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , ceph-devel@vger.kernel.org References: <20230329141354.516864-1-dhowells@redhat.com> <20230329141354.516864-38-dhowells@redhat.com> From: Xiubo Li In-Reply-To: <20230329141354.516864-38-dhowells@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org David, BTW, will this two patch depend on the others in this patch series ? I am planing to run a test with these two later. Thanks - Xiubo On 29/03/2023 22:13, David Howells wrote: > Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when > transmitting data. For the moment, this can only transmit one page at a > time because of the architecture of net/ceph/, but if > write_partial_message_data() can be given a bvec[] at a time by the > iteration code, this would allow pages to be sent in a batch. > > Signed-off-by: David Howells > cc: Ilya Dryomov > cc: Xiubo Li > cc: Jeff Layton > cc: "David S. Miller" > cc: Eric Dumazet > cc: Jakub Kicinski > cc: Paolo Abeni > cc: Jens Axboe > cc: Matthew Wilcox > cc: ceph-devel@vger.kernel.org > cc: netdev@vger.kernel.org > --- > net/ceph/messenger_v2.c | 89 +++++++++-------------------------------- > 1 file changed, 18 insertions(+), 71 deletions(-) > > diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c > index 301a991dc6a6..1637a0c21126 100644 > --- a/net/ceph/messenger_v2.c > +++ b/net/ceph/messenger_v2.c > @@ -117,91 +117,38 @@ static int ceph_tcp_recv(struct ceph_connection *con) > return ret; > } > > -static int do_sendmsg(struct socket *sock, struct iov_iter *it) > -{ > - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; > - int ret; > - > - msg.msg_iter = *it; > - while (iov_iter_count(it)) { > - ret = sock_sendmsg(sock, &msg); > - if (ret <= 0) { > - if (ret == -EAGAIN) > - ret = 0; > - return ret; > - } > - > - iov_iter_advance(it, ret); > - } > - > - WARN_ON(msg_data_left(&msg)); > - return 1; > -} > - > -static int do_try_sendpage(struct socket *sock, struct iov_iter *it) > -{ > - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; > - struct bio_vec bv; > - int ret; > - > - if (WARN_ON(!iov_iter_is_bvec(it))) > - return -EINVAL; > - > - while (iov_iter_count(it)) { > - /* iov_iter_iovec() for ITER_BVEC */ > - bvec_set_page(&bv, it->bvec->bv_page, > - min(iov_iter_count(it), > - it->bvec->bv_len - it->iov_offset), > - it->bvec->bv_offset + it->iov_offset); > - > - /* > - * sendpage cannot properly handle pages with > - * page_count == 0, we need to fall back to sendmsg if > - * that's the case. > - * > - * Same goes for slab pages: skb_can_coalesce() allows > - * coalescing neighboring slab objects into a single frag > - * which triggers one of hardened usercopy checks. > - */ > - if (sendpage_ok(bv.bv_page)) { > - ret = sock->ops->sendpage(sock, bv.bv_page, > - bv.bv_offset, bv.bv_len, > - CEPH_MSG_FLAGS); > - } else { > - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, bv.bv_len); > - ret = sock_sendmsg(sock, &msg); > - } > - if (ret <= 0) { > - if (ret == -EAGAIN) > - ret = 0; > - return ret; > - } > - > - iov_iter_advance(it, ret); > - } > - > - return 1; > -} > - > /* > * Write as much as possible. The socket is expected to be corked, > * so we don't bother with MSG_MORE/MSG_SENDPAGE_NOTLAST here. > * > * Return: > - * 1 - done, nothing (else) to write > + * >0 - done, nothing (else) to write > * 0 - socket is full, need to wait > * <0 - error > */ > static int ceph_tcp_send(struct ceph_connection *con) > { > + struct msghdr msg = { > + .msg_iter = con->v2.out_iter, > + .msg_flags = CEPH_MSG_FLAGS, > + }; > int ret; > > + if (WARN_ON(!iov_iter_is_bvec(&con->v2.out_iter))) > + return -EINVAL; > + > + if (con->v2.out_iter_sendpage) > + msg.msg_flags |= MSG_SPLICE_PAGES; > + > dout("%s con %p have %zu try_sendpage %d\n", __func__, con, > iov_iter_count(&con->v2.out_iter), con->v2.out_iter_sendpage); > - if (con->v2.out_iter_sendpage) > - ret = do_try_sendpage(con->sock, &con->v2.out_iter); > - else > - ret = do_sendmsg(con->sock, &con->v2.out_iter); > + > + ret = sock_sendmsg(con->sock, &msg); > + if (ret > 0) > + iov_iter_advance(&con->v2.out_iter, ret); > + else if (ret == -EAGAIN) > + ret = 0; > + > dout("%s con %p ret %d left %zu\n", __func__, con, ret, > iov_iter_count(&con->v2.out_iter)); > return ret; > -- Best Regards, Xiubo Li (李秀波) Email: xiubli@redhat.com/xiubli@ibm.com Slack: @Xiubo Li