Re: [PATCH] smb3: add rasize mount parameter to improve performance of readahead

From: Matthew Wilcox <willy@infradead.org>
To: Steve French <smfrench@gmail.com>
Cc: Shyam Prasad N <nspmangalore@gmail.com>,
	CIFS <linux-cifs@vger.kernel.org>,
	Jeff Layton <jlayton@redhat.com>,
	David Howells <dhowells@redhat.com>
Subject: Re: [PATCH] smb3: add rasize mount parameter to improve performance of readahead
Date: Sat, 1 May 2021 19:35:02 +0100	[thread overview]
Message-ID: <20210501183502.GU1847222@casper.infradead.org> (raw)
In-Reply-To: <CAH2r5mtE2g=p_rKThrDR_4N6=zqaBiz_KpK+bPpw5Q+qeFuTjQ@mail.gmail.com>

On Fri, Apr 30, 2021 at 02:22:20PM -0500, Steve French wrote:
> On Fri, Apr 30, 2021 at 7:00 AM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > On Fri, Apr 30, 2021 at 04:19:27PM +0530, Shyam Prasad N wrote:
> > > Although ideally, I feel that we (cifs.ko) should be able to read in
> > > larger granular "chunks" even for small reads, in expectation that
> > > surrounding offsets will be read soon.
> >
> > Why?  How is CIFS special and different from every other filesystem that
> > means you know what the access pattern of userspace is going to be better
> > than the generic VFS?
> 
> In general small chunks are bad for network file systems since the 'cost' of
> sending a large read or write on the network (and in the call stack on
> the client
> and server, with various task switches etc) is not much more than a small one.
> This can be different on a local file system with less latency between request
> and response and fewer task switches involved on client and server.

Block-based filesystems are often, but not always local.  For example,
we might be using nbd, iSCSI, FCoE or something similar to include
network latency between the filesystem and its storage.  Even without
those possibilities, a NAND SSD looks pretty similar.  Look at the
graphic titled "Idle Average Random Read Latency" on this page:

https://www.intel.ca/content/www/ca/en/architecture-and-technology/optane-technology/balancing-bandwidth-and-latency-article-brief.html

That seems to be showing 5us software latency for an SSD with 80us of
hardware latency.  That says to me we should have 16 outstanding reads
to a NAND SSD in order to keep the pipeline full.

Conversely, a network filesystem might be talking to localhost,
and seeing much lower latency compared to going across the data
center, between data centres or across the Pacific.

So, my point is that Linux's readahead is pretty poor.  Adding
hacks in for individual filesystems isn't a good route to fixing it,
and reading larger chunks has already passed the point of dimnishing
returns for many workloads.

I laid it out in a bit more detail here:
https://lore.kernel.org/linux-fsdevel/20210224155121.GQ2858050@casper.infradead.org/