Intermittent file corruption problems with cifs driver?

Intermittent file corruption problems with cifs driver?

[sean finney]

Hi all,

Recently at $customer I've been tasked into looking into a problem they
are intermittently having with corrupt file transfers from linux servers
to a windows share.  

Little info on the servers:

	Ubuntu Lucid 10.04
	Stock and up to date Linux 2.6.32-33-server distro package
	Stock cifs-utils 4.5-2 packages

Description of behavior:

	The servers are all part of a distributed service where each server
	regularly uploads 100-200MB zipfiles to the windows share.  Intermittently
	the resulting files will be corrupted.  On the client that performs
	the upload, the corrupted file will appear to have the correct checksum,
	but any other remote client will see it as corrupted.

	The problem used to be much more frequent, and mounting with -o directio
	seems to have greatly reduced, but not eliminated, the recurrence of the
	corruption.  But recently (perhaps due to higher reates of uploads?),
	the problem has started recurring.  It doesn't seem uniformly occuring,
	but rather in spurts where a couple files will be corrupted in one day,
	and then a week will go by with no corruptions.

	I do see occasional errors in the kernel logs, though I'm not sure if
	they are relevant or not (note that they're at substantially different
	times, and at present I have no way to correlate them with corruption,
	though I'm working on that):

	[170873.721023]  CIFS VFS: Error -104 sending data on socket to server
	[170873.728747]  CIFS VFS: Error -32 sending data on socket to server
	[515039.940104]  CIFS VFS: No response to cmd 115 mid 32714
	[515039.947933]  CIFS VFS: Send error in SessSetup = -11
	[521901.595381]  CIFS VFS: No response to cmd 46 mid 37426
	[521901.603422]  CIFS VFS: Send error in read = -11
	[2097744.571138]  CIFS VFS: No response for cmd 50 mid 48502
	[2097849.771138]  CIFS VFS: No response for cmd 114 mid 48519


Reading through the archives along with the rest of teh internetz I've found 
very little info.  Someone posted here back in february about a similar
sounding problem, though I do not see the wsize-len blocks of NULL bytes in
the resulting files like they did.

I've written a small python script that right now is running on a pair of
these servers, which with a couple dozen threads is uploading similarly sized
files of arbitrary data, and comparing the upload results of each other.
after a few hours I haven't seen it yet, but will keep it runnign for
a couple days to see if it shows up.

I've also found a couple suggestions out there to "disable linux
extensions" and "disable oplocks" when searching on the above kernel error
messages, but am hesitant to try them unless there's a strong indication
that they will help, and i'm not entirely sure if/whether they will.


does this ring a bell with anyone?  at this point i can't just do a
blanket "try the latest" upgrade of these servers because they're part
of a production application, at least without any further indication that
there was a fix for this problem between the current and latest versions.
If I can repro the problem, however, and can then take it to a non-prod
machine, then I might have a bit more flexibility, but in the meantime
thought I'd field the question here on the off chance...


thanks!
	sean

Re: Intermittent file corruption problems with cifs driver?

[Steve French]

On Mon, Sep 12, 2011 at 3:36 AM, sean finney <seanius@seanius.net> wrote:

> Hi all,
>
> Recently at $customer I've been tasked into looking into a problem they
> are intermittently having with corrupt file transfers from linux servers
> to a windows share.
>
> Little info on the servers:
>
>        Ubuntu Lucid 10.04
>        Stock and up to date Linux 2.6.32-33-server distro package
>        Stock cifs-utils 4.5-2 packages
>
> Description of behavior:
>
>        The servers are all part of a distributed service where each server
>        regularly uploads 100-200MB zipfiles to the windows share.
>  Intermittently
>        the resulting files will be corrupted.  On the client that performs
>        the upload, the corrupted file will appear to have the correct
> checksum,
>        but any other remote client will see it as corrupted.
>
>        The problem used to be much more frequent, and mounting with -o
> directio
>        seems to have greatly reduced, but not eliminated, the recurrence of
> the
>        corruption.  But recently (perhaps due to higher reates of
> uploads?),
>        the problem has started recurring.  It doesn't seem uniformly
> occuring,
>        but rather in spurts where a couple files will be corrupted in one
> day,
>        and then a week will go by with no corruptions.
>
>        I do see occasional errors in the kernel logs, though I'm not sure
> if
>        they are relevant or not (note that they're at substantially
> different
>        times, and at present I have no way to correlate them with
> corruption,
>        though I'm working on that):
>
>        [170873.721023]  CIFS VFS: Error -104 sending data on socket to
> server
>        [170873.728747]  CIFS VFS: Error -32 sending data on socket to
> server
>        [515039.940104]  CIFS VFS: No response to cmd 115 mid 32714
>        [515039.947933]  CIFS VFS: Send error in SessSetup = -11
>        [521901.595381]  CIFS VFS: No response to cmd 46 mid 37426
>        [521901.603422]  CIFS VFS: Send error in read = -11
>        [2097744.571138]  CIFS VFS: No response for cmd 50 mid 48502
>        [2097849.771138]  CIFS VFS: No response for cmd 114 mid 48519
>


With log entries like the above, the probability of having a file fail to
copy
is high (although I would expect it more with command 47 (SMB WriteAndX,
but presumably if you get an error reading (command 46) you will
also have the copy fail - depending on how the application handles
such errors).

The reconnection/retry behavior is much better in more current cifs,
especially when servers are sometimes very slow (as we saw
in clustered servers during occassional operations when cluster
overhead could cause > 15 second delays).   If the server
is taking greater than 15 seconds or so from time to time
(or as we see here with rc 104, if the server
or network randomly drops the connection), I would expect
big improvements in 2.6.39 or later (or equivalent
backport as some of the distros have done).

Disabling unix extensions (mount option "nounix") is
only going to make a difference to Samba and similar servers,
but I would not expect it to have an effect on this problem.
mounting "forcedirectio" could have an effect though as it
1) changes i/o sizes to more closely match the application
(since the cache is not being used)
2) allows the application to get errors on writes more
quickly (since some applications forget to check errors
on fsync or close, and expect write errors on the actual write)


>
>
> Reading through the archives along with the rest of teh internetz I've
> found
> very little info.  Someone posted here back in february about a similar
> sounding problem, though I do not see the wsize-len blocks of NULL bytes in
> the resulting files like they did.
>
> I've written a small python script that right now is running on a pair of
> these servers, which with a couple dozen threads is uploading similarly
> sized
> files of arbitrary data, and comparing the upload results of each other.
> after a few hours I haven't seen it yet, but will keep it runnign for
> a couple days to see if it shows up.
>
> I've also found a couple suggestions out there to "disable linux
> extensions" and "disable oplocks" when searching on the above kernel error
> messages, but am hesitant to try them unless there's a strong indication
> that they will help, and i'm not entirely sure if/whether they will.
>
>
> does this ring a bell with anyone?  at this point i can't just do a
> blanket "try the latest" upgrade of these servers because they're part
> of a production application, at least without any further indication that
> there was a fix for this problem between the current and latest versions.
> If I can repro the problem, however, and can then take it to a non-prod
> machine, then I might have a bit more flexibility, but in the meantime
> thought I'd field the question here on the off chance...
>
>
> thanks!
>        sean
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Thanks,

Steve

Fwd: Intermittent file corruption problems with cifs driver?

[Steve French]

On Mon, Sep 12, 2011 at 3:36 AM, sean finney <seanius-ADwgVSpYHhHR7s880joybQ@public.gmane.org> wrote:
>
> Hi all,
>
> Recently at $customer I've been tasked into looking into a problem they
> are intermittently having with corrupt file transfers from linux servers
> to a windows share.
>
> Little info on the servers:
>
>        Ubuntu Lucid 10.04
>        Stock and up to date Linux 2.6.32-33-server distro package
>        Stock cifs-utils 4.5-2 packages
>
> Description of behavior:
>
>        The servers are all part of a distributed service where each server
>        regularly uploads 100-200MB zipfiles to the windows share.  Intermittently
>        the resulting files will be corrupted.  On the client that performs
>        the upload, the corrupted file will appear to have the correct checksum,
>        but any other remote client will see it as corrupted.
>
>        The problem used to be much more frequent, and mounting with -o directio
>        seems to have greatly reduced, but not eliminated, the recurrence of the
>        corruption.  But recently (perhaps due to higher reates of uploads?),
>        the problem has started recurring.  It doesn't seem uniformly occuring,
>        but rather in spurts where a couple files will be corrupted in one day,
>        and then a week will go by with no corruptions.
>
>        I do see occasional errors in the kernel logs, though I'm not sure if
>        they are relevant or not (note that they're at substantially different
>        times, and at present I have no way to correlate them with corruption,
>        though I'm working on that):
>
>        [170873.721023]  CIFS VFS: Error -104 sending data on socket to server
>        [170873.728747]  CIFS VFS: Error -32 sending data on socket to server
>        [515039.940104]  CIFS VFS: No response to cmd 115 mid 32714
>        [515039.947933]  CIFS VFS: Send error in SessSetup = -11
>        [521901.595381]  CIFS VFS: No response to cmd 46 mid 37426
>        [521901.603422]  CIFS VFS: Send error in read = -11
>        [2097744.571138]  CIFS VFS: No response for cmd 50 mid 48502
>        [2097849.771138]  CIFS VFS: No response for cmd 114 mid 48519


With log entries like the above, the probability of having a file fail to copy
is high (although I would expect it more with command 47 (SMB WriteAndX,
but presumably if you get an error reading (command 46) you will
also have the copy fail - depending on how the application handles
such errors).

The reconnection/retry behavior is much better in more current cifs,
especially when servers are sometimes very slow (as we saw
in clustered servers during occassional operations when cluster
overhead could cause > 15 second delays).   If the server
is taking greater than 15 seconds or so from time to time
(or as we see here with rc 104, if the server
or network randomly drops the connection), I would expect
big improvements in 2.6.39 or later (or equivalent
backport as some of the distros have done).

Disabling unix extensions (mount option "nounix") is
only going to make a difference to Samba and similar servers,
but I would not expect it to have an effect on this problem.
mounting "forcedirectio" could have an effect though as it
1) changes i/o sizes to more closely match the application
(since the cache is not being used)
2) allows the application to get errors on writes more
quickly (since some applications forget to check errors
on fsync or close, and expect write errors on the actual write)

>
> Reading through the archives along with the rest of teh internetz I've found
> very little info.  Someone posted here back in february about a similar
> sounding problem, though I do not see the wsize-len blocks of NULL bytes in
> the resulting files like they did.
>
> I've written a small python script that right now is running on a pair of
> these servers, which with a couple dozen threads is uploading similarly sized
> files of arbitrary data, and comparing the upload results of each other.
> after a few hours I haven't seen it yet, but will keep it runnign for
> a couple days to see if it shows up.
>
> I've also found a couple suggestions out there to "disable linux
> extensions" and "disable oplocks" when searching on the above kernel error
> messages, but am hesitant to try them unless there's a strong indication
> that they will help, and i'm not entirely sure if/whether they will.
>
>
> does this ring a bell with anyone?  at this point i can't just do a
> blanket "try the latest" upgrade of these servers because they're part
> of a production application, at least without any further indication that
> there was a fix for this problem between the current and latest versions.
> If I can repro the problem, however, and can then take it to a non-prod
> machine, then I might have a bit more flexibility, but in the meantime
> thought I'd field the question here on the off chance...
>
>
> thanks!
>        sean
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Thanks,

Steve



--
Thanks,

Steve

Re: Intermittent file corruption problems with cifs driver?

[Jeff Layton]

On Mon, 12 Sep 2011 10:36:55 +0200
sean finney <seanius@seanius.net> wrote:

> Hi all,
> 
> Recently at $customer I've been tasked into looking into a problem they
> are intermittently having with corrupt file transfers from linux servers
> to a windows share.  
> 
> Little info on the servers:
> 
> 	Ubuntu Lucid 10.04
> 	Stock and up to date Linux 2.6.32-33-server distro package
> 	Stock cifs-utils 4.5-2 packages
> 
> Description of behavior:
> 
> 	The servers are all part of a distributed service where each server
> 	regularly uploads 100-200MB zipfiles to the windows share.  Intermittently
> 	the resulting files will be corrupted.  On the client that performs
> 	the upload, the corrupted file will appear to have the correct checksum,
> 	but any other remote client will see it as corrupted.
> 
> 	The problem used to be much more frequent, and mounting with -o directio
> 	seems to have greatly reduced, but not eliminated, the recurrence of the
> 	corruption.  But recently (perhaps due to higher reates of uploads?),
> 	the problem has started recurring.  It doesn't seem uniformly occuring,
> 	but rather in spurts where a couple files will be corrupted in one day,
> 	and then a week will go by with no corruptions.
> 
> 	I do see occasional errors in the kernel logs, though I'm not sure if
> 	they are relevant or not (note that they're at substantially different
> 	times, and at present I have no way to correlate them with corruption,
> 	though I'm working on that):
> 
> 	[170873.721023]  CIFS VFS: Error -104 sending data on socket to server
> 	[170873.728747]  CIFS VFS: Error -32 sending data on socket to server
> 	[515039.940104]  CIFS VFS: No response to cmd 115 mid 32714
> 	[515039.947933]  CIFS VFS: Send error in SessSetup = -11
> 	[521901.595381]  CIFS VFS: No response to cmd 46 mid 37426
> 	[521901.603422]  CIFS VFS: Send error in read = -11
> 	[2097744.571138]  CIFS VFS: No response for cmd 50 mid 48502
> 	[2097849.771138]  CIFS VFS: No response for cmd 114 mid 48519
> 
> 
> Reading through the archives along with the rest of teh internetz I've found 
> very little info.  Someone posted here back in february about a similar
> sounding problem, though I do not see the wsize-len blocks of NULL bytes in
> the resulting files like they did.
> 
> I've written a small python script that right now is running on a pair of
> these servers, which with a couple dozen threads is uploading similarly sized
> files of arbitrary data, and comparing the upload results of each other.
> after a few hours I haven't seen it yet, but will keep it runnign for
> a couple days to see if it shows up.
> 
> I've also found a couple suggestions out there to "disable linux
> extensions" and "disable oplocks" when searching on the above kernel error
> messages, but am hesitant to try them unless there's a strong indication
> that they will help, and i'm not entirely sure if/whether they will.
> 
> 
> does this ring a bell with anyone?  at this point i can't just do a
> blanket "try the latest" upgrade of these servers because they're part
> of a production application, at least without any further indication that
> there was a fix for this problem between the current and latest versions.
> If I can repro the problem, however, and can then take it to a non-prod
> machine, then I might have a bit more flexibility, but in the meantime
> thought I'd field the question here on the off chance...
> 
> 
> thanks!
> 	sean
> 


Older kernels were particularly bad about giving up on writes that
timed out. When this happens, it typically will mark the mapping bad
so that you get an error on fsync or close, but that's small
consolation since a lot of programs don't check the return value on
close.

That said, the messages you post above seem to indicate timeouts
on reads, not writes, but I don't recall whether writepages spewed any
errors when writes would time out.

The patchset that converted cifs to use async writes should not only
improve performance, but make this more robust as well. One thing you
can try is backporting 941b853 and see if that helps. Other than that
I'd suggest moving to a newer kernel.

Anything before 2.6.38 is probably going to suck for data integrity for
those reasons, unless someone backported the newer code to it (like we
did for RHEL6).

-- 
Jeff Layton <jlayton@redhat.com>

Re: Fwd: Intermittent file corruption problems with cifs driver?

[sean finney]

Hi Steve, Jeff,

Thanks for the detailed info and suggestions.  It sounds like a 3.0
backport is the way to go then, since I've seen a number of other fixes
and improvements go that way, let's see how that works :)


	sean

Re: Fwd: Intermittent file corruption problems with cifs driver?

[Steve French]

On Mon, Sep 12, 2011 at 9:37 AM, sean finney <seanius-ADwgVSpYHhHR7s880joybQ@public.gmane.org> wrote:
> Hi Steve, Jeff,
>
> Thanks for the detailed info and suggestions.  It sounds like a 3.0
> backport is the way to go then, since I've seen a number of other fixes
> and improvements go that way, let's see how that works :)

3.0 is a lot faster too



-- 
Thanks,

Steve