cifs large write performance improvements to Samba

cifs large write performance improvements to Samba

[Steve French]

Some informal results evaluating the results of new cifs directio mount
option and effects of changing the SMB network buffer size.  All tests
were run at least four times on each configuration and averaged.

Command executed: time dd if=/dev/zero of=/mnt/ofile bs=1M count=400 
(400MB copy to server with large writes)

MemTotal on system: 905148kB (from /proc/meminfo)

Kernel: current 2.6.10-rc3 from linux.bkbits.net/linux-2.5 bk
SLES9 JFS (and EXT3 on one test). Intel Pentium M, relatively slower IDE
drives (laptop). Kernel compiled for debug memory allocations.

CIFS filesystem version: 1.28

Samba version 3.0.10pre (current svn)

mount over loopback interface to samba server on the same box (to
maximize network transfer speed)


CIFS mount (default): 1 minute 46 seconds (constrained largely by the 4K
writes, due to no implementation of page coalescing and no
implementation of cifs_writepages)

CIFS mount (directio): 35.9 seconds (huge improvement due to now being
able to do up to 16K size writes to server, and reducing double caching
effects since page cache not used on client side). Running with a faster
processor (otherwise the same) this dropped to 20.8 second average.

CIFS mount (directio, insmod specifying larger CIFSMaxBufSize of 65024):
24.0 seconds  The larger buffer size (almost 4 times larger) led to
significant improvements (going beyond this to 120K network write size
did not improve performance on this test, actually got a bit worse).
Running with a faster processor (otherwise the same) this improved to
19.3 seconds average (and using the even larger 120K write size
performance dropped slightly to 20.6 seconds)

Additional comparison points - 
The older smbfs filesystem: 1 minute 7.04 seconds (note that smbfs read
performance is much worse than cifs in part due to lack of readpages
implementation in smbfs, but the writepage performances differ less
significantly)

Local performance: 
JFS  3.5 seconds
EXT3 12.8 seconds

EXT3 was somewhat slower than I expected.  

The does show that cifs directio mounts help a lot for large file copy -
and also that cifs badly needs an asyn writepages implementation (which
has obviously helped NFS on this kind of test). In addition CIFS needs
to reduce some of the CPU (one of the large buf allocs and memcpys in
the write path can be eliminated with some reasonably obvious changes). 
I doubt that enough analysis on the Samba server side has been done for
this particular case on 2.6 and that might yield some easy improvements
as well.  Also note that there was one case in which cifs to Samba ran
into the expected memory contention deadlock (client writes dirty page
to server, server blocked on memory allocation which is blocked on
client's write).

Re: cifs large write performance improvements to Samba

[bert hubert]

On Sun, Dec 12, 2004 at 11:45:38PM -0600, Steve French wrote:

> Running with a faster processor (otherwise the same) this improved to
> 19.3 seconds average (and using the even larger 120K write size
> performance dropped slightly to 20.6 seconds)

While this is all supremely cool, my inbox has several reactions from cifs
users about my never addressed query [1] on linux-cifs-clients regarding memory
leaks you can drive a truck through in cifs.

[1] http://marc.theaimsgroup.com/?l=linux-cifs-client&m=109221678006861&w=2

I'd love to deploy cifs and I've recommended it to many people, they've all
switched back to smbfs!

Now while I fully understand open source ethics (breaks, both pieces etc),
I'd really hope cifs would shape up in terms of ""enterprise reliability"".

Anything I can do to help, let me know. The referred URL contains some of my
conclusions at the time.

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: cifs large write performance improvements to Samba

[bert hubert]

On Mon, Dec 13, 2004 at 10:53:37AM -0600, Steve French wrote:
> 
> The current mainline (very recent 2.6.10-rc Linux tree) should be fine 
> from memory leak perspective.  No such leaks have been reported AFAIK on 
> current cifs code and certainly none that I have detected in heavy 
> stress testing. 

Indeed, thank you Steve! I had not noticed this progression from the
changelog messages. I've since verified on a multimillion file share that
cifs indeed shrinks the cifs_inode_cache under memory pressure. I've turned
on updatedb again, hope that the box survives it now.

> On the issue of regressing back to smbfs :)  There are a few things 
> which can be done that would help.

I've since moved some stuff back to cifs to see what happens. I'll try to
convince some other people so they can share/report their problems properly.

> 2) Public view of the status of testing - the raw data needs to be 
> posted regularly as kernel updated (and against five or six different 
> server types) so users see what is broken in smbfs (and so users can see 
> what posix issues are still being worked on cifs and any known 
> problems).   smbfs fails about half of the filesystem tests that I have 
> tried, due to stress issues, or because the tests requires better posix 
> compliance or because of various smbfs stability fixes.

That may be so but the definite perception is that cifs is unstable compared
to smbfs. Then again, this may have been related to out of memory conditions
which generaly tend to make things suck.

Thanks for your thoughtful reply, I'll see if I can provide useful feedback.

Regards,

bert

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: cifs large write performance improvements to Samba

[Steve French]

The current mainline (very recent 2.6.10-rc Linux tree) should be fine 
from memory leak perspective.  No such leaks have been reported AFAIK on 
current cifs code and certainly none that I have detected in heavy 
stress testing. 

I do need to add some additional filesystem tests to the regression test 
mix soon, and I have asked one  of the guys here  with a ppc64 box to 
run some more bigendian tests on it too.   The ltp has filesystems tests 
scatterred in multiple directories which I need to track down and setup 
scripts to automate (e.g. the sendfile tests are not in the same 
directory with other tests in testcases/kernel/fs, nor is the very 
useful "connectathon nfs" posix filesystem test suite)

The oops (referenced in your post) does need to be fixed of course, but 
since the code that would cause it is disabled (and only is broken to 
certain servers and is noted as broken in the TODO list, implying that 
it should not be turned on in /proc/fs/cifs) - I was considering it 
lower priority than the other issues recently fixed which have been 
consuming my time.  Fixing/adding support for extended security is 
getting closer to the top of the priority list now though.  If I can at 
least prevent the oops with a small change (even if it does not fully 
fix the spnego code) I will push that changeset in soon.   The userspace 
piece should be -- much -- easier to communicate with now that the 
kevents stuff is in.    Very high on the list as well is getting NTLMv2 
tested and working as many environments require it.  Figuring out the 
mysterious byte range lock failure which sometimes occurs on an unlock 
in locktest 7 (I noticed it starting after the nfs changes for the vfs 
posix locking a few months ago) and I have posted about before (Kernel 
panic - not syncing: Attempting to free lock with active blocklist) is a 
slightly higher priority.  Basically I need to figure out what is going 
on with the line in fs/locks.c?

       if (!list_empty <http://lxr.linux.no/ident?v=2.6.8.1;i=list_empty>(&fl->fl_block)
		 panic <http://lxr.linux.no/ident?v=2.6.8.1;i=panic>(/"Attempting to free lock with active block list"/);


Since I am not adding anything to the fl_block list intentionally, I 
need to find out what causes items to be added to the fl_block list (ie 
when the locks_insert_block and locks_delete_block call are made and why 
they sometimes happen differently in lock test 7 then in other byte 
range lock testcases in which unlock obviously works fine).

On the issue of regressing back to smbfs :)  There are a few things 
which can be done that would help.

1) Need to post an updated version of when to still use smbfs for an 
occassional mount (there are only a couple of cases in which smbfs has 
to be used now but they are important to some users - such as users who 
have to access an occassional old OS/2 or DOS server, or who need 
Kerberos), and I need to add it that chart to the fs/cifs/README and the 
project page etc.
2) Public view of the status of testing - the raw data needs to be 
posted regularly as kernel updated (and against five or six different 
server types) so users see what is broken in smbfs (and so users can see 
what posix issues are still being worked on cifs and any known 
problems).   smbfs fails about half of the filesystem tests that I have 
tried, due to stress issues, or because the tests requires better posix 
compliance or because of various smbfs stability fixes.

  If only someone could roll all of the key fs tests into a set of 
scripts which could generate one regularly updated set of test status 
chart ... one for each of XFS, JFS, ext3, Reiser3, CIFS (against various 
servers, Samba version etc), NFSv2, NFSv3, NFSv4 (against various 
servers), AFS but that would be a lot of work (not to run) but the first 
time writing/setup of the scripts to launch the tests in the right order 
since some failures may be expected (at least for the network 
filesystems) due to hard to implement features (missing fcntls, dnotify, 
get/setlease, differences in byte range lock semantics, lack of flock 
etc.) and also since the most sensible NFS, AFS and CIFS tests would 
involve more than one client (to test caching/oplock/token management 
semantics better) but no such fs tests AFAIK exist for Linux.

Re: cifs large write performance improvements to Samba

[cliff white]

On Mon, 13 Dec 2004 10:56:45 -0600
Steve French <smfrench@austin.rr.com> wrote:

> The current mainline (very recent 2.6.10-rc Linux tree) should be fine 
> from memory leak perspective.  No such leaks have been reported AFAIK on 
> current cifs code and certainly none that I have detected in heavy 
> stress testing. 
> 
> I do need to add some additional filesystem tests to the regression test 
> mix soon, and I have asked one  of the guys here  with a ppc64 box to 
> run some more bigendian tests on it too.   The ltp has filesystems tests 
> scatterred in multiple directories which I need to track down and setup 
> scripts to automate (e.g. the sendfile tests are not in the same 
> directory with other tests in testcases/kernel/fs, nor is the very 
> useful "connectathon nfs" posix filesystem test suite)
> 
> The oops (referenced in your post) does need to be fixed of course, but 
> since the code that would cause it is disabled (and only is broken to 
> certain servers and is noted as broken in the TODO list, implying that 
> it should not be turned on in /proc/fs/cifs) - I was considering it 
> lower priority than the other issues recently fixed which have been 
> consuming my time.  Fixing/adding support for extended security is 
> getting closer to the top of the priority list now though.  If I can at 
> least prevent the oops with a small change (even if it does not fully 
> fix the spnego code) I will push that changeset in soon.   The userspace 
> piece should be -- much -- easier to communicate with now that the 
> kevents stuff is in.    Very high on the list as well is getting NTLMv2 
> tested and working as many environments require it.  Figuring out the 
> mysterious byte range lock failure which sometimes occurs on an unlock 
> in locktest 7 (I noticed it starting after the nfs changes for the vfs 
> posix locking a few months ago) and I have posted about before (Kernel 
> panic - not syncing: Attempting to free lock with active blocklist) is a 
> slightly higher priority.  Basically I need to figure out what is going 
> on with the line in fs/locks.c?
> 
>        if (!list_empty <http://lxr.linux.no/ident?v=2.6.8.1;i=list_empty>(&fl->fl_block)
> 		 panic <http://lxr.linux.no/ident?v=2.6.8.1;i=panic>(/"Attempting to free lock with active block list"/);
> 
> 
> Since I am not adding anything to the fl_block list intentionally, I 
> need to find out what causes items to be added to the fl_block list (ie 
> when the locks_insert_block and locks_delete_block call are made and why 
> they sometimes happen differently in lock test 7 then in other byte 
> range lock testcases in which unlock obviously works fine).
> 
> On the issue of regressing back to smbfs :)  There are a few things 
> which can be done that would help.
> 
> 1) Need to post an updated version of when to still use smbfs for an 
> occassional mount (there are only a couple of cases in which smbfs has 
> to be used now but they are important to some users - such as users who 
> have to access an occassional old OS/2 or DOS server, or who need 
> Kerberos), and I need to add it that chart to the fs/cifs/README and the 
> project page etc.
> 2) Public view of the status of testing - the raw data needs to be 
> posted regularly as kernel updated (and against five or six different 
> server types) so users see what is broken in smbfs (and so users can see 
> what posix issues are still being worked on cifs and any known 
> problems).   smbfs fails about half of the filesystem tests that I have 
> tried, due to stress issues, or because the tests requires better posix 
> compliance or because of various smbfs stability fixes.
> 
>   If only someone could roll all of the key fs tests into a set of 
> scripts which could generate one regularly updated set of test status 
> chart ... one for each of XFS, JFS, ext3, Reiser3, CIFS (against various 
> servers, Samba version etc), NFSv2, NFSv3, NFSv4 (against various 
> servers), AFS but that would be a lot of work (not to run) but the first 
> time writing/setup of the scripts to launch the tests in the right order 
> since some failures may be expected (at least for the network 
> filesystems) due to hard to implement features (missing fcntls, dnotify, 
> get/setlease, differences in byte range lock semantics, lack of flock 
> etc.) and also since the most sensible NFS, AFS and CIFS tests would 
> involve more than one client (to test caching/oplock/token management 
> semantics better) but no such fs tests AFAIK exist for Linux.

We ( OSDL ) would be very interested in this sort of testing. We have some fs tests
wrappered currently
cliffw
OSDL
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb

Re: cifs large write performance improvements to Samba

[Steve French]

cliff white wrote:

>On Mon, 13 Dec 2004 10:56:45 -0600
>Steve French <smfrench@austin.rr.com> wrote:
>
>  If only someone could roll all of the key fs tests into a set of 
>scripts which could generate one regularly updated set of test status 
>chart ... one for each of XFS, JFS, ext3, Reiser3, CIFS (against various 
>servers, Samba version etc), NFSv2, NFSv3, NFSv4 (against various 
>servers), AFS but that would be a lot of work (not to run) but the first 
>time writing/setup of the scripts to launch the tests in the right order 
>since some failures may be expected (at least for the network 
>filesystems) due to hard to implement features (missing fcntls, dnotify, 
>get/setlease, differences in byte range lock semantics, lack of flock 
>etc.) and also since the most sensible NFS, AFS and CIFS tests would 
>involve more than one client (to test caching/oplock/token management 
>semantics better) but no such fs tests AFAIK exist for Linux.
>  
>
>
>We ( OSDL ) would be very interested in this sort of testing. We have some fs tests
>wrappered currently
>cliffw
>OSDL
>  
>

Generally what I wanted to see was:
1) at least one little endian and one big endian client to run the tests 
on (and for the network filesystems, at least one server).
2) execute a set of similar tests on each of the filesystems (for our 
Samba purposes e.g. ext3, xfs, jfs are particular important, and cifs 
and might as well run on nfsv3, nfsv4)
- fsx on each (specify an operation count of n=10000 should be sufficient)
- fsstress with at least 100 processes on each, with at least 2 loops, 
200 ops
- "connectathon nfs" tests on each. The local filesystems should all be 
able to pass these.
- for cifs to Windows servers, there is one of the "special" subcategory 
of tests that has to be skipped (since Windows server can not support 
the operation)
- and for cifs locktests 7 and 10 are expected to fail at this time
- dbench

There are others that can be run but they don't seem to broaden it much. 
What is very important to add into the mix - based on defects I have 
seen in various network and cluster filesystems over the past few years 
- are something similar to the following:
- Add a test which hits the three Linux specific fcntls (dnotify, set 
and get lease)
- Add a test which uses O_DIRECT open flag (could also simply use the 
mount flag for those filesystems which support that). NFS guys made 
trivial mod to fsx for this and ran with fsx -W -R (to disable mmapping 
when running with direct i/o)
- Add a test which exercises flock to the mix
- Add a maximum and minimium path name and path component test
- Add a test of creating directories with very large number of entries 
(cthon "basic" subtests can be easily modified for this).
- Add a test which does sendfile with data integrity checking from one 
process and a mix of normal and mmapped writes and reads from another 
set of processes
- Add a test of data integrity to the same network fs from multiple clients
- Add a test for stable nanosecond (or 100 nanosecond which is good 
enough for the network case) timestamps (ext2 and ext3 will fail this 
since they round timestamps down to the second when inode metadata is 
written out)
- A better test for the various O_ flags and file modes (especially 
important to run from multiple clients)
- A better byte range locking functional test
- xattr testing (maximum, minimum sizes, illegal requests, bad user 
buffers etc.) There is an xattr testcase in the ltp but have not 
analyzed it enough to see if it will do
- POSIX ACL testing (getfacl/setfacl)
- Trusted and Security attribute testing (to make sure the FS properly 
handles long attributes and/or values, short attributes and/or values, 
bad buffers etc.)

Re: cifs large write performance improvements to Samba

[Steve French]

cliff white wrote:

>
>> On Mon, 13 Dec 2004 10:56:45 -0600
>> Steve French <smfrench@austin.rr.com> wrote:
>>
>> If only someone could roll all of the key fs tests into a set of 
>> scripts which could generate one regularly updated set of test status 
>> chart ... one for each of XFS, JFS, ext3, Reiser3, CIFS (against 
>> various servers, Samba version etc), NFSv2, NFSv3, NFSv4 (against 
>> various servers), AFS but that would be a lot of work (not to run) 
>> but the first time writing/setup of the scripts to launch the tests 
>> in the right order since some failures may be expected (at least for 
>> the network filesystems) due to hard to implement features (missing 
>> fcntls, dnotify, get/setlease, differences in byte range lock 
>> semantics, lack of flock etc.) and also since the most sensible NFS, 
>> AFS and CIFS tests would involve more than one client (to test 
>> caching/oplock/token management semantics better) but no such fs 
>> tests AFAIK exist for Linux.
>>
>>
>>
>> We ( OSDL ) would be very interested in this sort of testing. We have 
>> some fs tests
>> wrappered currently
>> cliffw
>> OSDL
>>
>>
>
The other thing I forgot to mention ... we used to have a concept of 
"performance regression testing" (to make sure that we had not gotten a 
lot slower on the latest rc) - not just runs on every release candidate 
of a few complex benchmark tests (like SpecWeb or Netbench or some 
enterprise Java perf test) but the idea was to run on every rc an fs 
microbenchmark (more like iozone) to ensure that we did not have some 
small functional problem in an fs or mm subsystem was causing big, 
noticeable degradation in performance (large read or small read or large 
write or small write, random or sequential etc.). I have not seen anyone 
doing that on Linux in an automated fashion (e.g running iozone 
automated every time a new 2.6.x.rc on a half a dozen of the fs - simply 
to verify that things had not gotten drastically worse on a particular 
fs due to a bug or sideffect of a global VFS change).

Re: cifs large write performance improvements to Samba

[Marcelo Tosatti]

On Mon, Dec 13, 2004 at 12:43:27PM -0600, Steve French wrote:
> cliff white wrote:
> 
> >
> >>On Mon, 13 Dec 2004 10:56:45 -0600
> >>Steve French <smfrench@austin.rr.com> wrote:
> >>
> >>If only someone could roll all of the key fs tests into a set of 
> >>scripts which could generate one regularly updated set of test status 
> >>chart ... one for each of XFS, JFS, ext3, Reiser3, CIFS (against 
> >>various servers, Samba version etc), NFSv2, NFSv3, NFSv4 (against 
> >>various servers), AFS but that would be a lot of work (not to run) 
> >>but the first time writing/setup of the scripts to launch the tests 
> >>in the right order since some failures may be expected (at least for 
> >>the network filesystems) due to hard to implement features (missing 
> >>fcntls, dnotify, get/setlease, differences in byte range lock 
> >>semantics, lack of flock etc.) and also since the most sensible NFS, 
> >>AFS and CIFS tests would involve more than one client (to test 
> >>caching/oplock/token management semantics better) but no such fs 
> >>tests AFAIK exist for Linux.
> >>
> >>
> >>
> >>We ( OSDL ) would be very interested in this sort of testing. We have 
> >>some fs tests
> >>wrappered currently
> >>cliffw
> >>OSDL
> >>
> >>
> >
> The other thing I forgot to mention ... we used to have a concept of 
> "performance regression testing" (to make sure that we had not gotten a 
> lot slower on the latest rc) - not just runs on every release candidate 
> of a few complex benchmark tests (like SpecWeb or Netbench or some 
> enterprise Java perf test) but the idea was to run on every rc an fs 
> microbenchmark (more like iozone) to ensure that we did not have some 
> small functional problem in an fs or mm subsystem was causing big, 
> noticeable degradation in performance (large read or small read or large 
> write or small write, random or sequential etc.). I have not seen anyone 
> doing that on Linux in an automated fashion (e.g running iozone 
> automated every time a new 2.6.x.rc on a half a dozen of the fs - simply 
> to verify that things had not gotten drastically worse on a particular 
> fs due to a bug or sideffect of a global VFS change).

Yes, we definately need that.

The STP framework is running different benchmarks on each v2.6 new -mm and -mainline 
release already. Take a look at 

http://www.osdl.org/lab_activities/kernel_testing/stp/search.lnk/search_test_requests

There are two things considered important to me:

- Need to have a wide set of benchmarks to simulate a wide set of workloads. Having 
one microbenchmark limits the usefulness of the tests. This is very important.

- Need to parse the results of such tests in a way thats they are easily visualized

The second item of course depends on the number of variations that are ran. For
example you mention several filesystem, that would increase the amount of results
 dramatically.

We also need the manpower to carefully interpret the results (which is not trivial).

I would like to see different variations of memory size and CPU (1-2-4-8 CPU variation 
is already being done by STP), have been asking Cliff for such functionality for 
the past days.

One unfortunate thing is the failure rate for STP is currently really high. 
Cliff, the results I've received today and yesterday have a rough 50% failure rate.
Is anyone investigating that?

But Steve, to resume, the STP guys are working hard to make such 
IO/VM/FS performance tests automated and useful for the community. 

We need to push'em harder! :) 

Re: cifs large write performance improvements to Samba

[Hans Reiser]

Marcelo Tosatti wrote:

>
>> I have not seen anyone 
>>doing that on Linux in an automated fashion (e.g running iozone 
>>automated every time a new 2.6.x.rc on a half a dozen of the fs - simply 
>>to verify that things had not gotten drastically worse on a particular 
>>fs due to a bug or sideffect of a global VFS change).
>>    
>>
>
>Yes, we definately need that.
>
>  
>
Andrew Morton is saying that iozone does things real apps don't do, that 
is, it dirties mmap'd pages enough to swamp the machine.

Do you guys agree or disagree with that?

Reiser4 needs iozone optimization work which we haven't bothered with yet.

Re: cifs large write performance improvements to Samba

[Marcelo Tosatti]

On Thu, Dec 16, 2004 at 10:58:07AM -0800, Hans Reiser wrote:
> Marcelo Tosatti wrote:
> 
> >
> >>I have not seen anyone 
> >>doing that on Linux in an automated fashion (e.g running iozone 
> >>automated every time a new 2.6.x.rc on a half a dozen of the fs - simply 
> >>to verify that things had not gotten drastically worse on a particular 
> >>fs due to a bug or sideffect of a global VFS change).
> >>   
> >>
> >
> >Yes, we definately need that.
> >
> > 
> >
> Andrew Morton is saying that iozone does things real apps don't do, that 
> is, it dirties mmap'd pages enough to swamp the machine.
> 
> Do you guys agree or disagree with that?
> 
> Reiser4 needs iozone optimization work which we haven't bothered with yet.

I'm not really familiar with iozone's behaviour, sorry. 

Steve, Andrew?

PS: Yep, another important part (to me at least) of all this automated performance 
testing effort is a detailed understanding of the tests being performed.