From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D23E6C433DB for ; Tue, 16 Feb 2021 05:18:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2B66964DDA for ; Tue, 16 Feb 2021 05:18:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2B66964DDA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 65F888D0151; Tue, 16 Feb 2021 00:18:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 611418D0140; Tue, 16 Feb 2021 00:18:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AF108D0151; Tue, 16 Feb 2021 00:18:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0119.hostedemail.com [216.40.44.119]) by kanga.kvack.org (Postfix) with ESMTP id 32BDA8D0140 for ; Tue, 16 Feb 2021 00:18:16 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E58C718021872 for ; Tue, 16 Feb 2021 05:18:15 +0000 (UTC) X-FDA: 77822974950.29.act09_3614c1b27641 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id AF8AD180FDDAD for ; Tue, 16 Feb 2021 05:18:15 +0000 (UTC) X-HE-Tag: act09_3614c1b27641 X-Filterd-Recvd-Size: 6870 Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Tue, 16 Feb 2021 05:18:15 +0000 (UTC) Received: by mail-lf1-f42.google.com with SMTP id f1so13987730lfu.3 for ; Mon, 15 Feb 2021 21:18:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wujt3rEkfz1gBVeqiGJuebMiBQ6EXLfMvV1N3ToSLEg=; b=G1pqEFCRklKjZBFj6/WMCJvTupO2n0FY8ZdI0lPq/s4scqtvILGzYRjptwygMdmpRa M+KeFCamCX7+3s612uofPbUoy+TvLFU2pQ6phK0LV2NAzy93/pvgpyeDNCKXu1m3TIsR /JRe1NQ7zecm8IoIWOnai/+Mb1HWXFbRLmwpW94ujHQvIiP4qK1vE9bzvHL3luJjXbCX v/3dxDs4i1sCs7N3E+GWC4A0+6hExVYA1awT07sNClmS58kanpx9GZxOEdpkTmEh67Sn MWdV18swd+PP8ozVqBEaOcsJb8YvbLp79/4tz2d3bqmg33RlROI3BkfhaKA8qUFbErCF pzEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wujt3rEkfz1gBVeqiGJuebMiBQ6EXLfMvV1N3ToSLEg=; b=W+ihMIS0U1jpzGeQgTCPZ4IBBaUz6beqczuc70fD4pR3mZQdHP/z7ECzvabzDkkFQn GkShy9S8Gm3Aw6N+1v0woa3iBxelxD5y1RSB6N+MhCpX6CWc9ay5h9143SJS8BqP82H7 6SlN5kzD9E3S6JbThP0PUlfvZ/GDY/JzQX9+SeTkmW70JHdvkqE/LVfiN1Q35/USm1Aj s/tVlQ1zQR7qLYNKgsbmQIJOxMG2nrqW0d/+vRGWB2sPDsbjCVdBZoofCSwX+CoV1HAv tuQWRoHZpr+nuMwX5f6MXwOHILcPYGrBWQ4DvLDdYKTMP3umI8zPDiwXJKsi4YSdaIyw y3HQ== X-Gm-Message-State: AOAM530e2lLIsy3yH2+691L9jmZI7dTw46/unrsafhSPTcObPYZwQ3mT r+YUBaOx0uDhPqBejAHO6Vgvp2LscaIMCaYk2HQ= X-Google-Smtp-Source: ABdhPJy9H3EXUc8XuYPIQKDxwM3oPFRhR4c+QWVWNLoFZd2YMqS0S6y9PDhVsr2YqLR65ivGiOSLLFSUYk3dT5a6PGE= X-Received: by 2002:a05:6512:2118:: with SMTP id q24mr4994475lfr.133.1613452693694; Mon, 15 Feb 2021 21:18:13 -0800 (PST) MIME-Version: 1.0 References: <161340385320.1303470.2392622971006879777.stgit@warthog.procyon.org.uk> <9e49f96cd80eaf9c8ed267a7fbbcb4c6467ee790.camel@redhat.com> <20210216021015.GH2858050@casper.infradead.org> In-Reply-To: <20210216021015.GH2858050@casper.infradead.org> From: Steve French Date: Mon, 15 Feb 2021 23:18:02 -0600 Message-ID: Subject: Re: [PATCH 00/33] Network fs helper library & fscache kiocb API [ver #3] To: Matthew Wilcox Cc: Jeff Layton , David Howells , Trond Myklebust , Anna Schumaker , Steve French , Dominique Martinet , CIFS , ceph-devel@vger.kernel.org, linux-cachefs@redhat.com, Alexander Viro , linux-mm , linux-afs@lists.infradead.org, v9fs-developer@lists.sourceforge.net, Christoph Hellwig , linux-fsdevel , linux-nfs , Linus Torvalds , David Wysochanski , LKML , William Kucharski Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Feb 15, 2021 at 8:10 PM Matthew Wilcox wrote: > > On Mon, Feb 15, 2021 at 06:40:27PM -0600, Steve French wrote: > > It could be good if netfs simplifies the problem experienced by > > network filesystems on Linux with readahead on large sequential reads > > - where we don't get as much parallelism due to only having one > > readahead request at a time (thus in many cases there is 'dead time' > > on either the network or the file server while waiting for the next > > readpages request to be issued). This can be a significant > > performance problem for current readpages when network latency is long > > (or e.g. in cases when network encryption is enabled, and hardware > > offload not available so time consuming on the server or client to > > encrypt the packet). > > > > Do you see netfs much faster than currentreadpages for ceph? > > > > Have you been able to get much benefit from throttling readahead with > > ceph from the current netfs approach for clamping i/o? > > The switch from readpages to readahead does help in a couple of corner > cases. For example, if you have two processes reading the same file at > the same time, one will now block on the other (due to the page lock) > rather than submitting a mess of overlapping and partial reads. > > We're not there yet on having multiple outstanding reads. Bill and I > had a chat recently about how to make the readahead code detect that > it is in a "long fat pipe" situation (as opposed to just dealing with > a slow device), and submit extra readahead requests to make best use of > the bandwidth and minimise blocking of the application. > > That's not something for the netfs code to do though; we can get into > that situation with highly parallel SSDs. This (readahead behavior improvements in Linux, on single large file sequential read workloads like cp or grep) gets particularly interesting with SMB3 as multichannel becomes more common. With one channel having one readahead request pending on the network is suboptimal - but not as bad as when multichannel is negotiated. Interestingly in most cases two network connections to the same server (different TCP sockets,but the same mount, even in cases where only network adapter) can achieve better performance - but still significantly lags Windows (and probably other clients) as in Linux we don't keep multiple I/Os in flight at one time (unless different files are being read at the same time by different threads). As network adapters are added and removed from the server (other client typically poll to detect interface changes, and SMB3 also leverages the "witness protocol" to get notification of adapter additions or removals) - it would be helpful to change the maximum number of readahead requests in flight. In addition, as the server throttles back (reducing the number of 'credits' granted to the client) it will be important to give hints to the readahead logic about reducing the number of read ahead requests in flight. Keeping multiple readahead requests is easier to imagine when multiple processes are copying or reading files, but there are many scenarios where we could do better with parallelizing a single process doing copy by ensuring that there is no 'dead time' on the network. -- Thanks, Steve