* Re: [PATCH] cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+
[not found] <CAEUGjKiLPQP9wp0AgLUvHgKBOe9We2a-RQaZ7cd7CvhnarwWiw@mail.gmail.com>
@ 2020-04-10 18:25 ` Pavel Shilovsky
[not found] ` <CAEUGjKhSBNQboKOMFMgos9OQfxcLQZsXp8aBrUSFcaSe1saH2Q@mail.gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Pavel Shilovsky @ 2020-04-10 18:25 UTC (permalink / raw)
To: Jones Syue; +Cc: linux-cifs, Samba Technical, Kernel Mailing List
пт, 10 апр. 2020 г. в 09:50, Jones Syue via samba-technical
<samba-technical@lists.samba.org>:
>
> Hello list,
>
> please help review whether the attached patch is correct, thank you.
>
> Found a read performance issue when linux kernel page size is 64KB.
> If linux kernel page size is 64KB and mount options cache=strict &
> vers=2.1+,
> it does not support cifs_readpages(). Instead, it is using cifs_readpage()
> and
> cifs_read() with maximum read IO size 16KB, which is much slower than read
> IO
> size 1MB when negotiated SMB 2.1+. Since modern SMB server supported SMB
> 2.1+
> and Max Read Size can reach more than 64KB (for example 1MB ~ 8MB), this
> patch
> do one more check on max_read to determine whether server support
> readpages()
> and improve read performance for page size 64KB & cache=strict & vers=2.1+.
>
> The client is a linux box with linux kernel 4.2.8,
> page size 64KB (CONFIG_ARM64_64K_PAGES=y),
> cpu arm 1.7GHz, and use mount.cifs as smb client.
> The server is another linux box with linux kernel 4.2.8,
> share a file '10G.img' with size 10GB,
> and use samba-4.7.12 as smb server.
>
> The client mount a share from the server with different
> cache options: cache=strict and cache=none,
> mount -tcifs //<server_ip>/Public /cache_strict
> -overs=3.0,cache=strict,username=<xxx>,password=<yyy>
> mount -tcifs //<server_ip>/Public /cache_none
> -overs=3.0,cache=none,username=<xxx>,password=<yyy>
>
> The client download a 10GbE file from the server accross 1GbE network,
> dd if=/cache_strict/10G.img of=/dev/null bs=1M count=10240
> dd if=/cache_none/10G.img of=/dev/null bs=1M count=10240
>
> Found that cache=strict (without patch) is slower read throughput and
> smaller
> read IO size than cache=none.
> cache=strict (without patch): read throughput 40MB/s, read IO size is 16KB
> cache=strict (with patch): read throughput 113MB/s, read IO size is 1MB
> cache=none: read throughput 109MB/s, read IO size is 1MB
>
> Looks like if page size is 64KB,
> cifs_set_ops() would use cifs_addr_ops_smallbuf instead of cifs_addr_ops,
>
> /* check if server can support readpages */
> if (cifs_sb_master_tcon(cifs_sb)->ses->server->maxBuf <
> PAGE_SIZE + MAX_CIFS_HDR_SIZE)
> inode->i_data.a_ops = &cifs_addr_ops_smallbuf;
> else
> inode->i_data.a_ops = &cifs_addr_ops;
>
> maxBuf is came from 2 places, SMB2_negotiate() and CIFSSMBNegotiate(),
> (SMB2_MAX_BUFFER_SIZE is 64KB)
> SMB2_negotiate():
> /* set it to the maximum buffer size value we can send with 1 credit */
> server->maxBuf = min_t(unsigned int, le32_to_cpu(rsp->MaxTransactSize),
> SMB2_MAX_BUFFER_SIZE);
> CIFSSMBNegotiate():
> server->maxBuf = le32_to_cpu(pSMBr->MaxBufferSize);
>
> Page size 64KB and cache=strict lead to read_pages() use cifs_readpage()
> instead
> of cifs_readpages(), and then cifs_read() using maximum read IO size 16KB,
> which is much slower than maximum read IO size 1MB.
> (CIFSMaxBufSize is 16KB by default)
>
> /* FIXME: set up handlers for larger reads and/or convert to async */
> rsize = min_t(unsigned int, cifs_sb->rsize, CIFSMaxBufSize);
>
Hi Jones,
Thanks for the patch!
It will work although it is probably a little bit cleaner to
initialize server->max_read to server->maxBuf for SMB1 and use the
server->max_read in the readpages condition check instead.
@Others, thoughts?
--
Best regards,
Pavel Shilovsky
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+
[not found] ` <CAEUGjKhSBNQboKOMFMgos9OQfxcLQZsXp8aBrUSFcaSe1saH2Q@mail.gmail.com>
@ 2020-04-13 16:37 ` Pavel Shilovsky
2020-04-14 1:13 ` Steve French
2020-04-14 19:58 ` Steve French
1 sibling, 1 reply; 6+ messages in thread
From: Pavel Shilovsky @ 2020-04-13 16:37 UTC (permalink / raw)
To: Jones Syue; +Cc: linux-cifs, Samba Technical, Kernel Mailing List
Looks good, thanks!
Reviewed-by: Pavel Shilovsky <pshilov@microsoft.com>
--
Best regards,
Pavel Shilovsky
вс, 12 апр. 2020 г. в 19:23, Jones Syue <jonessyue@qnap.com>:
>
> Hello Pavel
>
> Thanks for kindly reviewing!
> Please find the attached v2.patch.
>
> --
> Regards,
> Jones Syue | 薛懷宗
> QNAP Systems, Inc.
>
>
> On Sat, Apr 11, 2020 at 2:25 AM Pavel Shilovsky <piastryyy@gmail.com> wrote:
> >
> > Hi Jones,
> >
> > Thanks for the patch!
> >
> > It will work although it is probably a little bit cleaner to
> > initialize server->max_read to server->maxBuf for SMB1 and use the
> > server->max_read in the readpages condition check instead.
> >
> > @Others, thoughts?
> >
> > --
> > Best regards,
> > Pavel Shilovsky
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+
2020-04-13 16:37 ` Pavel Shilovsky
@ 2020-04-14 1:13 ` Steve French
0 siblings, 0 replies; 6+ messages in thread
From: Steve French @ 2020-04-14 1:13 UTC (permalink / raw)
To: Pavel Shilovsky
Cc: Jones Syue, linux-cifs, Samba Technical, Kernel Mailing List
merged into cifs-2.6.git for-next pending testing
On Mon, Apr 13, 2020 at 11:39 AM Pavel Shilovsky via samba-technical
<samba-technical@lists.samba.org> wrote:
>
> Looks good, thanks!
>
> Reviewed-by: Pavel Shilovsky <pshilov@microsoft.com>
>
> --
> Best regards,
> Pavel Shilovsky
>
> вс, 12 апр. 2020 г. в 19:23, Jones Syue <jonessyue@qnap.com>:
> >
> > Hello Pavel
> >
> > Thanks for kindly reviewing!
> > Please find the attached v2.patch.
> >
> > --
> > Regards,
> > Jones Syue | 薛懷宗
> > QNAP Systems, Inc.
> >
> >
> > On Sat, Apr 11, 2020 at 2:25 AM Pavel Shilovsky <piastryyy@gmail.com> wrote:
> > >
> > > Hi Jones,
> > >
> > > Thanks for the patch!
> > >
> > > It will work although it is probably a little bit cleaner to
> > > initialize server->max_read to server->maxBuf for SMB1 and use the
> > > server->max_read in the readpages condition check instead.
> > >
> > > @Others, thoughts?
> > >
> > > --
> > > Best regards,
> > > Pavel Shilovsky
>
--
Thanks,
Steve
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+
[not found] ` <CAEUGjKhSBNQboKOMFMgos9OQfxcLQZsXp8aBrUSFcaSe1saH2Q@mail.gmail.com>
2020-04-13 16:37 ` Pavel Shilovsky
@ 2020-04-14 19:58 ` Steve French
[not found] ` <CAEUGjKhpgmhj9RzcGQXPuFUyoqsUnk2d3oCpOYBdR=EwCO21YQ@mail.gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Steve French @ 2020-04-14 19:58 UTC (permalink / raw)
To: Jones Syue
Cc: Pavel Shilovsky, linux-cifs, Samba Technical, Kernel Mailing List
Did you also test (at least briefly) with vers=1.0 since some of your
code affects that code path too?
And if anyone figures out how to configure an x86_64 Linux to use
PAGE_SIZE of 64K or larger let me know...
On Sun, Apr 12, 2020 at 9:24 PM Jones Syue via samba-technical
<samba-technical@lists.samba.org> wrote:
>
> Hello Pavel
>
> Thanks for kindly reviewing!
> Please find the attached v2.patch.
>
> --
> Regards,
> Jones Syue | 薛懷宗
> QNAP Systems, Inc.
>
>
> On Sat, Apr 11, 2020 at 2:25 AM Pavel Shilovsky <piastryyy@gmail.com> wrote:
> >
> > Hi Jones,
> >
> > Thanks for the patch!
> >
> > It will work although it is probably a little bit cleaner to
> > initialize server->max_read to server->maxBuf for SMB1 and use the
> > server->max_read in the readpages condition check instead.
> >
> > @Others, thoughts?
> >
> > --
> > Best regards,
> > Pavel Shilovsky
--
Thanks,
Steve
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+
[not found] ` <CAEUGjKhpgmhj9RzcGQXPuFUyoqsUnk2d3oCpOYBdR=EwCO21YQ@mail.gmail.com>
@ 2020-04-16 7:33 ` Jones Syue
2020-04-27 2:49 ` Jones Syue
1 sibling, 0 replies; 6+ messages in thread
From: Jones Syue @ 2020-04-16 7:33 UTC (permalink / raw)
To: Steve French
Cc: Pavel Shilovsky, linux-cifs, Samba Technical, Kernel Mailing List
Hello Steve
> Test read performance over 1GbE network with command:
Also test read performance over 10GbE network,
vers=2.1+ can reach over 600 MB/s with v2.patch.
aarch64, page size 64KB (CONFIG_ARM64_64K_PAGES=y), linux-4.2.8,
cpu Annapurna Labs Alpine AL324 Quad-core ARM Cortex-A57 CPU @ 1.70GHz,
ram 8GB,
with patch,
vers=1.0,cache=strict: read throughput 110MB/s, max read IO size 16KB
vers=2.0,cache=strict: read throughput 106MB/s, max read IO size 16KB
vers=2.1,cache=strict: read throughput 667MB/s, max read IO size 1MB
vers=3.0,cache=strict: read throughput 639MB/s, max read IO size 1MB
without patch,
vers=1.0,cache=strict: read throughput 107MB/s, max read IO size 16KB
vers=2.0,cache=strict: read throughput 107MB/s, max read IO size 16KB
vers=2.1,cache=strict: read throughput 106MB/s, max read IO size 16KB
vers=3.0,cache=strict: read throughput 106MB/s, max read IO size 16KB
command:
mount -tcifs //<server_ip>/<share> /remote_strict
-overs=<x.y>,cache=strict,username=<uu>,password=<pp>
dd if=/remote_strict/10G.img of=/dev/null bs=1M count=10240
--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.
On Thu, Apr 16, 2020 at 11:46 AM Jones Syue <jonessyue@qnap.com> wrote:
>
> Hello Steve
>
> > Did you also test (at least briefly) with vers=1.0 since some of your
> > code affects that code path too?
>
> Yes test v2.patch on 2 platforms aarch64 (page size 64KB) and x86_64
> (page size 4KB), vers=1.0 read function works fine on both.
>
> Test read performance over 1GbE network with command:
> 'dd if=/remote_strict/10G.img of=/dev/null bs=1M count=10240'
>
> For read performance on aarch64 (page size 64KB), vers=[1.0|2.0] is not as
> fast as vers=2.1+, max_read on both SMB 1 (16KB) and SMB 2.0 (64KB) are
> still smaller then page size 64KB plus packet header size, hence do not
> support readpages.
> aarch64, page size 64KB (CONFIG_ARM64_64K_PAGES=y), linux-4.2.8,
> cpu Annapurna Labs Alpine AL324 Quad-core ARM Cortex-A57 CPU @ 1.70GHz,
> ram 8GB,
> with patch,
> vers=1.0,cache=strict: read throughput 40MB/s, max read IO size 16KB
> vers=2.0,cache=strict: read throughput 40MB/s, max read IO size 16KB
> vers=2.1,cache=strict: read throughput 115MB/s, max read IO size 1MB
> vers=3.0,cache=strict: read throughput 115MB/s, max read IO size 1MB
> without patch,
> vers=1.0,cache=strict: read throughput 40MB/s, max read IO size 16KB
> vers=2.0,cache=strict: read throughput 40MB/s, max read IO size 16KB
> vers=2.1,cache=strict: read throughput 40MB/s, max read IO size 16KB
> vers=3.0,cache=strict: read throughput 40MB/s, max read IO size 16KB
>
> For read performance on x86_64 (page size 4KB), all vers can support
> readpages because max_read is bigger than page size 4KB plus packet header
> size.
> x86_64, page size 4KB, linux-4.2.8,
> cpu AMD Embedded R-Series RX-421ND 2.10GHz,
> ram 4GB,
> without patch,
> vers=1.0,cache=strict: read throughput 109MB/s, read IO size 60KB
> vers=2.0,cache=strict: read throughput 115MB/s, read IO size 64KB
> vers=2.1,cache=strict: read throughput 117MB/s, read IO size 1MB
> vers=3.0,cache=strict: read throughput 117MB/s, read IO size 1MB
> with patch,
> vers=1.0,cache=strict: read throughput 110MB/s, read IO size 60KB
> vers=2.0,cache=strict: read throughput 115MB/s, read IO size 64KB
> vers=2.1,cache=strict: read throughput 117MB/s, read IO size 1MB
> vers=3.0,cache=strict: read throughput 117MB/s, read IO size 1MB
>
> > And if anyone figures out how to configure an x86_64 Linux to use
> > PAGE_SIZE of 64K or larger let me know...
> I am using physical platform with arm cpu and aarch64 toolchain,
> perhaps try qemu-system-aarch64 later.
>
> --
> Regards,
> Jones Syue | 薛懷宗
> QNAP Systems, Inc.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+
[not found] ` <CAEUGjKhpgmhj9RzcGQXPuFUyoqsUnk2d3oCpOYBdR=EwCO21YQ@mail.gmail.com>
2020-04-16 7:33 ` Jones Syue
@ 2020-04-27 2:49 ` Jones Syue
1 sibling, 0 replies; 6+ messages in thread
From: Jones Syue @ 2020-04-27 2:49 UTC (permalink / raw)
To: linux-cifs
Cc: Pavel Shilovsky, Steve French, Samba Technical, Kernel Mailing List
> > And if anyone figures out how to configure an x86_64 Linux to use
> > PAGE_SIZE of 64K or larger let me know...
> I am using physical platform with arm cpu and aarch64 toolchain,
> perhaps try qemu-system-aarch64 later.
For reference using qemu-system-aarch64 + linux-5.6.4 + 64KB page to test
cifs read, this patch can improve cifs read performance:
with patch: read throughput 39 MB/s, SMB read IO size 4MB
/ # dd if=/mnt/cifs/1G.img of=/dev/null bs=4M count=256
256+0 records in
256+0 records out
1073741824 bytes (1.0GB) copied, 25.982352 seconds, 39.4MB/s
[~] # strace -p 23934
sendfile(38, 32, [297795584] => [301989888], 4194304) = 4194304
without patch: read throughput 18 MB/s, SMB read IO size 16KB
/ # dd if=/mnt/cifs/1G.img of=/dev/null bs=4M count=256G
256+0 records in
256+0 records out
1073741824 bytes (1.0GB) copied, 54.367686 seconds, 18.8MB/s
[~] <0> strace -p 15786
sendfile(38, 32, [452984832] => [453001216], 16384) = 16384
This link is a easy way to compile aarch64 linux kernel with page size 64KB
, a simple rootfs with busybox, and run it on qemu-system-aarch64:
https://docs.google.com/document/d/1NSVd-dib_asugCZHmZgohLZXHxV25ftzYtUDSppY3hA/edit?usp=sharing
--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-04-27 2:49 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAEUGjKiLPQP9wp0AgLUvHgKBOe9We2a-RQaZ7cd7CvhnarwWiw@mail.gmail.com>
2020-04-10 18:25 ` [PATCH] cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+ Pavel Shilovsky
[not found] ` <CAEUGjKhSBNQboKOMFMgos9OQfxcLQZsXp8aBrUSFcaSe1saH2Q@mail.gmail.com>
2020-04-13 16:37 ` Pavel Shilovsky
2020-04-14 1:13 ` Steve French
2020-04-14 19:58 ` Steve French
[not found] ` <CAEUGjKhpgmhj9RzcGQXPuFUyoqsUnk2d3oCpOYBdR=EwCO21YQ@mail.gmail.com>
2020-04-16 7:33 ` Jones Syue
2020-04-27 2:49 ` Jones Syue
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.