CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Tim Perry]

We have a windows 2003 server with a directory shared. I connect to it from linux (Ubuntu 12.04) using CIFS. Once mounted, I can read/write files, list files, et cetera from the linux box. However, if I run "find . -name \*txt" and hit ctrl-c before the find command finishes then the mount immediately fails (the find command takes about 15 minutes to complete on this particular file system). Not only that, once the mount fails I can't successfully remount the drive without rebooting the linux box. This makes me think a kernel bug is in play.... Occasionally using lsof or fuser I can find a process that is keeping the mount alive, kill it, and remount the filesystem without rebooting. However, this is rare.

Perhaps I'm just missing some setting to CIFS, but I can't figure it out. I know what you are thinking "Use the google, Luke" which I have tried. However, I get about 5000 hits for "I kan't mount my Windozes on my linux, kan u give m3 teh codez pleaze" which is almost, but not quite my issue.

Anyhow, /etc/fstab contains:
//server/All_proj /server/all_proj cifs sec=ntlmv2i,iocharset=utf8,uid=perry,credentials=/home/me/.smbcred,noauto,filemode=0777,dir_mode=0777,_netdev,gid=5000 0 0

Around the time the mount fails, I see this in /var/log/syslog:
Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
....

I also see the following in /var/log/kern.log
Dec 13 10:01:29 chinstrap kernel: [   66.567463] CIFS VFS: default security mechanism requested.  The default security mechanism will be upgraded from ntlm to ntlmv2 in kernel release 3.3
Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
....

So....I know there must be somewhere to report this as a bug and beg for help, but I have no idea where? Do I go to the Ubuntu folks at Canonical? To the Linux kernel team? Here?

Help please!

Thanks,
Tim

Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Jeff Layton]

On Tue, 18 Dec 2012 15:16:56 -0800 (PST)
Tim Perry <tdparmor-sambabugs-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:

> We have a windows 2003 server with a directory shared. I connect to it from linux (Ubuntu 12.04) using CIFS. Once mounted, I can read/write files, list files, et cetera from the linux box. However, if I run "find . -name \*txt" and hit ctrl-c before the find command finishes then the mount immediately fails (the find command takes about 15 minutes to complete on this particular file system). Not only that, once the mount fails I can't successfully remount the drive without rebooting the linux box. This makes me think a kernel bug is in play.... Occasionally using lsof or fuser I can find a process that is keeping the mount alive, kill it, and remount the filesystem without rebooting. However, this is rare.
> 
> Perhaps I'm just missing some setting to CIFS, but I can't figure it out. I know what you are thinking "Use the google, Luke" which I have tried. However, I get about 5000 hits for "I kan't mount my Windozes on my linux, kan u give m3 teh codez pleaze" which is almost, but not quite my issue.
> 
> Anyhow, /etc/fstab contains:
> //server/All_proj /server/all_proj cifs sec=ntlmv2i,iocharset=utf8,uid=perry,credentials=/home/me/.smbcred,noauto,filemode=0777,dir_mode=0777,_netdev,gid=5000 0 0
> 
> Around the time the mount fails, I see this in /var/log/syslog:
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
> ....
> 
> I also see the following in /var/log/kern.log
> Dec 13 10:01:29 chinstrap kernel: [   66.567463] CIFS VFS: default security mechanism requested.  The default security mechanism will be upgraded from ntlm to ntlmv2 in kernel release 3.3
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
> ....
> 
> So....I know there must be somewhere to report this as a bug and beg for help, but I have no idea where? Do I go to the Ubuntu folks at Canonical? To the Linux kernel team? Here?
> 

I have no idea what kernel 12.04 uses. Can you tell us the kernel
version? As far as I know, Ubuntu generally pulls their kernels
directly from upstream ones (or the stable series) so this is probably
a problem for mainline kernels too.

It sounds like signals are somehow causing the sequence numbers for
signatures to get out-of-whack. -13 is -EACCES, so it sounds like the
server is just returning errors and not disconnecting the socket when
this happens. That behavior varies between different windows versions...

The reason you need to reboot to clear this is probably because
something is holding a reference to the connection to the server. When
you try to unmount and remount it, then it ends up reusing the same
socket which is obviously still having problems.

The best thing you could do is to provide as simple a way as possible
to reproduce this problem. You may also want to do some investigation
on your own by stracing the process.

Is it getting hung on a particular syscall before you ever issue your
signal? If not, then does it get hung on a syscall afterward? What
state is the process in at that time?

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Tim Perry]

Dear Jeff, et. al.,

I can reproduce the problem by starting "find . -name \*.ext"and killing it when connected to either of our two Windows 2003 Servers. I can *not* reproduce it doing the same thing connected to a windows 7 box.

$ uname -a
Linux servername 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012 i686 i686 i386 GNU/Linux
$ cat /proc/version

Linux version 3.2.0-34-generic (buildd@roseapple) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.1 LTS
Release:        12.04
Codename:       precise


I tried using strace but hitting ctrl-c killed strace (obviously, oops), but interestingly, this did *not* hang the file system. I will try and kill the find command (kill -9 perhaps?) and see if I can recreate the error that way.

Tim








----- Original Message -----
From: Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: Tim Perry <tdparmor-sambabugs-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
Cc: "linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" <linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Sent: Wednesday, December 19, 2012 6:03 AM
Subject: Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

On Tue, 18 Dec 2012 15:16:56 -0800 (PST)
Tim Perry <tdparmor-sambabugs-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:

> We have a windows 2003 server with a directory shared. I connect to it from linux (Ubuntu 12.04) using CIFS. Once mounted, I can read/write files, list files, et cetera from the linux box. However, if I run "find . -name \*txt" and hit ctrl-c before the find command finishes then the mount immediately fails (the find command takes about 15 minutes to complete on this particular file system). Not only that, once the mount fails I can't successfully remount the drive without rebooting the linux box. This makes me think a kernel bug is in play.... Occasionally using lsof or fuser I can find a process that is keeping the mount alive, kill it, and remount the filesystem without rebooting. However, this is rare.
> 
> Perhaps I'm just missing some setting to CIFS, but I can't figure it out. I know what you are thinking "Use the google, Luke" which I have tried. However, I get about 5000 hits for "I kan't mount my Windozes on my linux, kan u give m3 teh codez pleaze" which is almost, but not quite my issue.
> 
> Anyhow, /etc/fstab contains:
> //server/All_proj /server/all_proj cifs sec=ntlmv2i,iocharset=utf8,uid=perry,credentials=/home/me/.smbcred,noauto,filemode=0777,dir_mode=0777,_netdev,gid=5000 0 0
> 
> Around the time the mount fails, I see this in /var/log/syslog:
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
> ....
> 
> I also see the following in /var/log/kern.log
> Dec 13 10:01:29 chinstrap kernel: [   66.567463] CIFS VFS: default security mechanism requested.  The default security mechanism will be upgraded from ntlm to ntlmv2 in kernel release 3.3
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
> ....
> 
> So....I know there must be somewhere to report this as a bug and beg for help, but I have no idea where? Do I go to the Ubuntu folks at Canonical? To the Linux kernel team? Here?
> 

I have no idea what kernel 12.04 uses. Can you tell us the kernel
version? As far as I know, Ubuntu generally pulls their kernels
directly from upstream ones (or the stable series) so this is probably
a problem for mainline kernels too.

It sounds like signals are somehow causing the sequence numbers for
signatures to get out-of-whack. -13 is -EACCES, so it sounds like the
server is just returning errors and not disconnecting the socket when
this happens. That behavior varies between different windows versions...

The reason you need to reboot to clear this is probably because
something is holding a reference to the connection to the server. When
you try to unmount and remount it, then it ends up reusing the same
socket which is obviously still having problems.

The best thing you could do is to provide as simple a way as possible
to reproduce this problem. You may also want to do some investigation
on your own by stracing the process.

Is it getting hung on a particular syscall before you ever issue your
signal? If not, then does it get hung on a syscall afterward? What
state is the process in at that time?

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Fw: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Tim Perry]

Dear Jeff, et. al.,


I can reproduce the problem by starting "find . -name \*.ext"and killing it when connected to either of our two Windows 2003 Servers. I can *not* reproduce it doing the same thing connected to a windows 7 box.

$ uname -a
Linux servername 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012 i686 i686 i386 GNU/Linux
$ cat /proc/version

Linux version 3.2.0-34-generic (buildd@roseapple) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.1 LTS
Release:        12.04
Codename:       precise


I tried using strace but hitting ctrl-c killed strace (obviously, oops), but interestingly, this did *not* hang the file system. I will try and kill the find command (kill -9 perhaps?) and see if I can recreate the error that way.

CONTINUING HERE:
I don't think strace on the find command will help because it isn't making the network connections. CIFS is making the network connections. Maybe I can cause the mount to happen with an strace version of CIFS?  How would I do that?

Anyhow, I opened two terminal windows and proceeded as follows:

In terminal 1:

$ strace find . -name \*adzzz >& ~/straceFind.txt


In terminal 2:
$ ps aux | grep find | grep -v strace
perry     2583 12.6  0.0   4792  1088 pts/5    R+   11:27   0:00 find . -name *adzzz
perry     2585  0.0  0.0   4388   828 pts/2    S+   11:27   0:00 grep find
$ kill -9 2583

File system dies.

I've attaced the straceFind.txt, but it just shows find walking the filesystem tree:
statat64(AT_FDCWD, "0010", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "0010", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
fchdir(5)                               = 0
getdents64(5, /* 14 entries */, 32768)  = 448
getdents64(5, /* 0 entries */, 32768)   = 0
close(5)                                = 0
fstatat64(AT_FDCWD, "_vti_cnf", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "_vti_cnf", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
fchdir(5)                               = 0
getdents64(5, /* 13 entries */, 32768)  = 416
getdents64(5, /* 0 entries */, 32768)   = 0
close(5)                                = 0
open("..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_NOFOLLOW) = 5
fstat64(5, {st_mode=S_IFDIR|0777, st_size=0, ...}) = 0
fchdir(


Ideas?

Tim








----- Original Message -----
From: Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: Tim Perry <tdparmor-sambabugs-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
Cc: "linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" <linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Sent: Wednesday, December 19, 2012 6:03 AM
Subject: Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

On Tue, 18 Dec 2012 15:16:56 -0800 (PST)
Tim Perry <tdparmor-sambabugs-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:

> We have a windows 2003 server with a directory shared. I connect to it from linux (Ubuntu 12.04) using CIFS. Once mounted, I can read/write files, list files, et cetera from the linux box. However, if I run "find . -name \*txt" and hit ctrl-c before the find command finishes then the mount immediately fails (the find command takes about 15 minutes to complete on this particular file system). Not only that, once the mount fails I can't successfully remount the drive without rebooting the linux box. This makes me think a kernel bug is in play.... Occasionally using lsof or fuser I can find a process that is keeping the mount alive, kill it, and remount the filesystem without rebooting. However, this is rare.
> 
> Perhaps I'm just missing some setting to CIFS, but I can't figure it out. I know what you are thinking "Use the google, Luke" which I have tried. However, I get about 5000 hits for "I kan't mount my Windozes on my linux, kan u give m3 teh codez pleaze" which is almost, but not quite my issue.
> 
> Anyhow, /etc/fstab contains:
> //server/All_proj /server/all_proj cifs sec=ntlmv2i,iocharset=utf8,uid=perry,credentials=/home/me/.smbcred,noauto,filemode=0777,dir_mode=0777,_netdev,gid=5000 0 0
> 
> Around the time the mount fails, I see this in /var/log/syslog:
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
> ....
> 
> I also see the following in /var/log/kern.log
> Dec 13 10:01:29 chinstrap kernel: [   66.567463] CIFS VFS: default security mechanism requested.  The default security mechanism will be upgraded from ntlm to ntlmv2 in kernel release 3.3
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028311] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028318] CIFS VFS: Send error in Close = -13
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028558] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.028886] CIFS VFS: Unexpected SMB signature
> Dec 13 10:40:35 chinstrap kernel: [ 2412.029395] CIFS VFS: Unexpected SMB signature
> ....
> 
> So....I know there must be somewhere to report this as a bug and beg for help, but I have no idea where? Do I go to the Ubuntu folks at Canonical? To the Linux kernel team? Here?
> 

I have no idea what kernel 12.04 uses. Can you tell us the kernel
version? As far as I know, Ubuntu generally pulls their kernels
directly from upstream ones (or the stable series) so this is probably
a problem for mainline kernels too.

It sounds like signals are somehow causing the sequence numbers for
signatures to get out-of-whack. -13 is -EACCES, so it sounds like the
server is just returning errors and not disconnecting the socket when
this happens. That behavior varies between different windows versions...

The reason you need to reboot to clear this is probably because
something is holding a reference to the connection to the server. When
you try to unmount and remount it, then it ends up reusing the same
socket which is obviously still having problems.

The best thing you could do is to provide as simple a way as possible
to reproduce this problem. You may also want to do some investigation
on your own by stracing the process.

Is it getting hung on a particular syscall before you ever issue your
signal? If not, then does it get hung on a syscall afterward? What
state is the process in at that time?

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Jeff Layton]

On Wed, 19 Dec 2012 11:30:32 -0800 (PST)
Tim Perry <tim.perry-kUnrLEE8I/oKpCjwLQ55aZ9NZdITTVap@public.gmane.org> wrote:

> Dear Jeff, et. al.,
> 
> 
> I can reproduce the problem by starting "find . -name \*.ext"and killing it when connected to either of our two Windows 2003 Servers. I can *not* reproduce it doing the same thing connected to a windows 7 box.
> 
> $ uname -a
> Linux servername 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012 i686 i686 i386 GNU/Linux
> $ cat /proc/version
> 
> Linux version 3.2.0-34-generic (buildd@roseapple) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012
> $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:    Ubuntu 12.04.1 LTS
> Release:        12.04
> Codename:       precise
> 
> 
> I tried using strace but hitting ctrl-c killed strace (obviously, oops), but interestingly, this did *not* hang the file system. I will try and kill the find command (kill -9 perhaps?) and see if I can recreate the error that way.
> 
> CONTINUING HERE:
> I don't think strace on the find command will help because it isn't making the network connections. CIFS is making the network connections. Maybe I can cause the mount to happen with an strace version of CIFS?  How would I do that?
> 
> Anyhow, I opened two terminal windows and proceeded as follows:
> 
> In terminal 1:
> 
> $ strace find . -name \*adzzz >& ~/straceFind.txt
> 
> 
> In terminal 2:
> $ ps aux | grep find | grep -v strace
> perry     2583 12.6  0.0   4792  1088 pts/5    R+   11:27   0:00 find . -name *adzzz
> perry     2585  0.0  0.0   4388   828 pts/2    S+   11:27   0:00 grep find
> $ kill -9 2583
> 
> File system dies.
> 
> I've attaced the straceFind.txt, but it just shows find walking the filesystem tree:
> statat64(AT_FDCWD, "0010", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
> openat(AT_FDCWD, "0010", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
> fchdir(5)                               = 0
> getdents64(5, /* 14 entries */, 32768)  = 448
> getdents64(5, /* 0 entries */, 32768)   = 0
> close(5)                                = 0
> fstatat64(AT_FDCWD, "_vti_cnf", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
> openat(AT_FDCWD, "_vti_cnf", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
> fchdir(5)                               = 0
> getdents64(5, /* 13 entries */, 32768)  = 416
> getdents64(5, /* 0 entries */, 32768)   = 0
> close(5)                                = 0
> open("..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_NOFOLLOW) = 5
> fstat64(5, {st_mode=S_IFDIR|0777, st_size=0, ...}) = 0
> fchdir(
> 
> 
> Ideas?
> 

That kernel is pretty old, so you may want to try a more recent one.

You may first want to start by tracing with wireshark -- see what's
happening on the wire before and after the signal is delivered.

If it works against win7 then it's likely that win7 disconnects the
socket when the signatures are wrong. With that, we'd reestablish the
connection and things would start working again. I suspect that win2k8
just starts returning an error that we map to -EACCES.

It's possible that we should disconnect the client when the signatures
start looking wrong, but I think we need to understand why signals are
causing this issue in the first place.

There are some places where we do interruptible sleeps (vs. killable
ones). It's possible that SIGINT (which is what ^c generally delivers)
is causing havok there.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Jeff Layton]

On Thu, 20 Dec 2012 09:38:06 -0500
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Wed, 19 Dec 2012 11:30:32 -0800 (PST)
> Tim Perry <tim.perry-kUnrLEE8I/oKpCjwLQ55aZ9NZdITTVap@public.gmane.org> wrote:
> 
> > Dear Jeff, et. al.,
> > 
> > 
> > I can reproduce the problem by starting "find . -name \*.ext"and killing it when connected to either of our two Windows 2003 Servers. I can *not* reproduce it doing the same thing connected to a windows 7 box.
> > 
> > $ uname -a
> > Linux servername 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012 i686 i686 i386 GNU/Linux
> > $ cat /proc/version
> > 
> > Linux version 3.2.0-34-generic (buildd@roseapple) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012
> > $ lsb_release -a
> > No LSB modules are available.
> > Distributor ID: Ubuntu
> > Description:    Ubuntu 12.04.1 LTS
> > Release:        12.04
> > Codename:       precise
> > 
> > 
> > I tried using strace but hitting ctrl-c killed strace (obviously, oops), but interestingly, this did *not* hang the file system. I will try and kill the find command (kill -9 perhaps?) and see if I can recreate the error that way.
> > 
> > CONTINUING HERE:
> > I don't think strace on the find command will help because it isn't making the network connections. CIFS is making the network connections. Maybe I can cause the mount to happen with an strace version of CIFS?  How would I do that?
> > 
> > Anyhow, I opened two terminal windows and proceeded as follows:
> > 
> > In terminal 1:
> > 
> > $ strace find . -name \*adzzz >& ~/straceFind.txt
> > 
> > 
> > In terminal 2:
> > $ ps aux | grep find | grep -v strace
> > perry     2583 12.6  0.0   4792  1088 pts/5    R+   11:27   0:00 find . -name *adzzz
> > perry     2585  0.0  0.0   4388   828 pts/2    S+   11:27   0:00 grep find
> > $ kill -9 2583
> > 
> > File system dies.
> > 
> > I've attaced the straceFind.txt, but it just shows find walking the filesystem tree:
> > statat64(AT_FDCWD, "0010", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
> > openat(AT_FDCWD, "0010", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
> > fchdir(5)                               = 0
> > getdents64(5, /* 14 entries */, 32768)  = 448
> > getdents64(5, /* 0 entries */, 32768)   = 0
> > close(5)                                = 0
> > fstatat64(AT_FDCWD, "_vti_cnf", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
> > openat(AT_FDCWD, "_vti_cnf", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
> > fchdir(5)                               = 0
> > getdents64(5, /* 13 entries */, 32768)  = 416
> > getdents64(5, /* 0 entries */, 32768)   = 0
> > close(5)                                = 0
> > open("..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_NOFOLLOW) = 5
> > fstat64(5, {st_mode=S_IFDIR|0777, st_size=0, ...}) = 0
> > fchdir(
> > 
> > 
> > Ideas?
> > 
> 
> That kernel is pretty old, so you may want to try a more recent one.
> 
> You may first want to start by tracing with wireshark -- see what's
> happening on the wire before and after the signal is delivered.
> 
> If it works against win7 then it's likely that win7 disconnects the
> socket when the signatures are wrong. With that, we'd reestablish the
> connection and things would start working again. I suspect that win2k8
> just starts returning an error that we map to -EACCES.
> 
> It's possible that we should disconnect the client when the signatures
> start looking wrong, but I think we need to understand why signals are
> causing this issue in the first place.
> 
> There are some places where we do interruptible sleeps (vs. killable
> ones). It's possible that SIGINT (which is what ^c generally delivers)
> is causing havok there.
> 

I had a look at the code today and suspect that I know what the problem
is. When the kernel goes to send a request, it first signs it and then
bumps the sequence numbers that it tracks. If the request doesn't
actually make it out onto the wire, like when the task catches a
signal, those sequence numbers remain high even though the request
didn't go out.

Here's an untested patch that might help tell whether this is the
case. You may want to try it and see if it does. Note that this fix is
a bit of a kludge and is not suitable for merging!

A better fix would involve changing when the sequence number gets
bumped in the first place. If this patch seems to help things, then
I'll look at coding up that up.

diff --git a/fs/cifs/transport.c b/fs/cifs/transport.c
index 76d974c..4520234 100644
--- a/fs/cifs/transport.c
+++ b/fs/cifs/transport.c
@@ -334,10 +334,14 @@ uncork:
 		server->tcpStatus = CifsNeedReconnect;
 	}
 
-	if (rc < 0 && rc != -EINTR)
-		cERROR(1, "Error %d sending data on socket to server", rc);
-	else
+	if (rc < 0) {
+		if (rc == -EINTR)
+			server->sequence_number -= 2;
+		else
+			cERROR(1, "Error %d sending data on socket to server", rc);
+	} else {
 		rc = 0;
+	}
 
 	return rc;
 }

Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Jeff Layton]

On Sun, 23 Dec 2012 09:10:34 -0500
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Thu, 20 Dec 2012 09:38:06 -0500
> Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> 
> > On Wed, 19 Dec 2012 11:30:32 -0800 (PST)
> > Tim Perry <tim.perry-kUnrLEE8I/oKpCjwLQ55aZ9NZdITTVap@public.gmane.org> wrote:
> > 
> > > Dear Jeff, et. al.,
> > > 
> > > 
> > > I can reproduce the problem by starting "find . -name \*.ext"and killing it when connected to either of our two Windows 2003 Servers. I can *not* reproduce it doing the same thing connected to a windows 7 box.
> > > 
> > > $ uname -a
> > > Linux servername 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012 i686 i686 i386 GNU/Linux
> > > $ cat /proc/version
> > > 
> > > Linux version 3.2.0-34-generic (buildd@roseapple) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012
> > > $ lsb_release -a
> > > No LSB modules are available.
> > > Distributor ID: Ubuntu
> > > Description:    Ubuntu 12.04.1 LTS
> > > Release:        12.04
> > > Codename:       precise
> > > 
> > > 
> > > I tried using strace but hitting ctrl-c killed strace (obviously, oops), but interestingly, this did *not* hang the file system. I will try and kill the find command (kill -9 perhaps?) and see if I can recreate the error that way.
> > > 
> > > CONTINUING HERE:
> > > I don't think strace on the find command will help because it isn't making the network connections. CIFS is making the network connections. Maybe I can cause the mount to happen with an strace version of CIFS?  How would I do that?
> > > 
> > > Anyhow, I opened two terminal windows and proceeded as follows:
> > > 
> > > In terminal 1:
> > > 
> > > $ strace find . -name \*adzzz >& ~/straceFind.txt
> > > 
> > > 
> > > In terminal 2:
> > > $ ps aux | grep find | grep -v strace
> > > perry     2583 12.6  0.0   4792  1088 pts/5    R+   11:27   0:00 find . -name *adzzz
> > > perry     2585  0.0  0.0   4388   828 pts/2    S+   11:27   0:00 grep find
> > > $ kill -9 2583
> > > 
> > > File system dies.
> > > 
> > > I've attaced the straceFind.txt, but it just shows find walking the filesystem tree:
> > > statat64(AT_FDCWD, "0010", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
> > > openat(AT_FDCWD, "0010", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
> > > fchdir(5)                               = 0
> > > getdents64(5, /* 14 entries */, 32768)  = 448
> > > getdents64(5, /* 0 entries */, 32768)   = 0
> > > close(5)                                = 0
> > > fstatat64(AT_FDCWD, "_vti_cnf", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
> > > openat(AT_FDCWD, "_vti_cnf", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 5
> > > fchdir(5)                               = 0
> > > getdents64(5, /* 13 entries */, 32768)  = 416
> > > getdents64(5, /* 0 entries */, 32768)   = 0
> > > close(5)                                = 0
> > > open("..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_NOFOLLOW) = 5
> > > fstat64(5, {st_mode=S_IFDIR|0777, st_size=0, ...}) = 0
> > > fchdir(
> > > 
> > > 
> > > Ideas?
> > > 
> > 
> > That kernel is pretty old, so you may want to try a more recent one.
> > 
> > You may first want to start by tracing with wireshark -- see what's
> > happening on the wire before and after the signal is delivered.
> > 
> > If it works against win7 then it's likely that win7 disconnects the
> > socket when the signatures are wrong. With that, we'd reestablish the
> > connection and things would start working again. I suspect that win2k8
> > just starts returning an error that we map to -EACCES.
> > 
> > It's possible that we should disconnect the client when the signatures
> > start looking wrong, but I think we need to understand why signals are
> > causing this issue in the first place.
> > 
> > There are some places where we do interruptible sleeps (vs. killable
> > ones). It's possible that SIGINT (which is what ^c generally delivers)
> > is causing havok there.
> > 
> 
> I had a look at the code today and suspect that I know what the problem
> is. When the kernel goes to send a request, it first signs it and then
> bumps the sequence numbers that it tracks. If the request doesn't
> actually make it out onto the wire, like when the task catches a
> signal, those sequence numbers remain high even though the request
> didn't go out.
> 
> Here's an untested patch that might help tell whether this is the
> case. You may want to try it and see if it does. Note that this fix is
> a bit of a kludge and is not suitable for merging!
> 
> A better fix would involve changing when the sequence number gets
> bumped in the first place. If this patch seems to help things, then
> I'll look at coding up that up.
> 
> diff --git a/fs/cifs/transport.c b/fs/cifs/transport.c
> index 76d974c..4520234 100644
> --- a/fs/cifs/transport.c
> +++ b/fs/cifs/transport.c
> @@ -334,10 +334,14 @@ uncork:
>  		server->tcpStatus = CifsNeedReconnect;
>  	}
>  
> -	if (rc < 0 && rc != -EINTR)
> -		cERROR(1, "Error %d sending data on socket to server", rc);
> -	else
> +	if (rc < 0) {
> +		if (rc == -EINTR)
> +			server->sequence_number -= 2;
> +		else
> +			cERROR(1, "Error %d sending data on socket to server", rc);
> +	} else {
>  		rc = 0;
> +	}
>  
>  	return rc;
>  }
> 
> 

I was able to reproduce this, and I don't think the above patch will
fix it (at least not completely). The problem seems to be that the NT
cancel command is screwing up the sequence numbers. We'll have to do
some research to figure out why that's occurring.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

On Mon, 2012-12-24 at 09:14 -0500, Jeff Layton wrote:
> On Sun, 23 Dec 2012 09:10:34 -0500
> Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
[...]
> > I had a look at the code today and suspect that I know what the problem
> > is. When the kernel goes to send a request, it first signs it and then
> > bumps the sequence numbers that it tracks. If the request doesn't
> > actually make it out onto the wire, like when the task catches a
> > signal, those sequence numbers remain high even though the request
> > didn't go out.
> > 
> > Here's an untested patch that might help tell whether this is the
> > case. You may want to try it and see if it does. Note that this fix is
> > a bit of a kludge and is not suitable for merging!
> > 
> > A better fix would involve changing when the sequence number gets
> > bumped in the first place. If this patch seems to help things, then
> > I'll look at coding up that up.
[...]
> I was able to reproduce this, and I don't think the above patch will
> fix it (at least not completely). The problem seems to be that the NT
> cancel command is screwing up the sequence numbers. We'll have to do
> some research to figure out why that's occurring.

Jeff, we got a bug report in Debian which seems to be the same problem:
<http://bugs.debian.org/695492>.  Please cc John Darrah and the bug
address as above.

Ben.

-- 
Ben Hutchings
It is easier to change the specification to fit the program than vice versa.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: CIFS mount fails if I ctrl-c a long-running find process (Linux mounting Windows share)

[Jeff Layton]

On Sat, 29 Dec 2012 01:24:36 +0100
Ben Hutchings <ben-/+tVBieCtBitmTQ+vhA3Yw@public.gmane.org> wrote:

> On Mon, 2012-12-24 at 09:14 -0500, Jeff Layton wrote:
> > On Sun, 23 Dec 2012 09:10:34 -0500
> > Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> [...]
> > > I had a look at the code today and suspect that I know what the problem
> > > is. When the kernel goes to send a request, it first signs it and then
> > > bumps the sequence numbers that it tracks. If the request doesn't
> > > actually make it out onto the wire, like when the task catches a
> > > signal, those sequence numbers remain high even though the request
> > > didn't go out.
> > > 
> > > Here's an untested patch that might help tell whether this is the
> > > case. You may want to try it and see if it does. Note that this fix is
> > > a bit of a kludge and is not suitable for merging!
> > > 
> > > A better fix would involve changing when the sequence number gets
> > > bumped in the first place. If this patch seems to help things, then
> > > I'll look at coding up that up.
> [...]
> > I was able to reproduce this, and I don't think the above patch will
> > fix it (at least not completely). The problem seems to be that the NT
> > cancel command is screwing up the sequence numbers. We'll have to do
> > some research to figure out why that's occurring.
> 
> Jeff, we got a bug report in Debian which seems to be the same problem:
> <http://bugs.debian.org/695492>.  Please cc John Darrah and the bug
> address as above.
> 
> Ben.
> 

You may want to try this patch. It seems to fix the problem for me, but
I think there is probably some more work to do in this area.

http://www.spinics.net/lists/linux-cifs/msg07576.html

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

mount.cifs error code docs?

[Matthew M. DeLoera]

Hello,

I've been searching to find a list of return codes from mount.cifs, with no luck. They're not listed in the manpage. I can google individual errors, but I'm hoping for something like a list of all possible errors. Or at least the most common ones.

Short of looking at code, is there any possibility that someone's documented details on something like causes for error code 5? Or other return values?

Any suggestions would be most appreciated!

Best regards,
- Matthew--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mount.cifs error code docs?

[Jeff Layton]

On Wed, 13 Mar 2013 10:00:10 -0400
"Matthew M. DeLoera" <mdeloera-CERpylIU+L8AvxtiuMwx3w@public.gmane.org> wrote:

> Hello,
> 
> I've been searching to find a list of return codes from mount.cifs, with no luck. They're not listed in the manpage. I can google individual errors, but I'm hoping for something like a list of all possible errors. Or at least the most common ones.
> 
> Short of looking at code, is there any possibility that someone's documented details on something like causes for error code 5? Or other return values?
> 
> Any suggestions would be most appreciated!
> 
> Best regards,
> - Matthew--

Like most mount helpers, the return codes are typically what the kernel
returns on the mount() syscall. Error code '5' is likely EIO, which is
a very generic error.

Cranking up debugging may give you more info as to what went wrong:

    https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: mount.cifs error code docs?

[Matthew M. DeLoera]

Jeff,

Thanks for the info. I should clarify that it's not that I've got an error. I'm trying to document expected error codes for my project.

Though I am doing some testing to assemble at least some error cases and associated codes. Agreed about error 5 in this case, though was hoping there might be some sort of list out there somewhere about at least some of the various CIFS failures that get rolled together under error 5.

Best regards,
- Matthew


On Mar 13, 2013, at 10:23 AM, Jeff Layton wrote:

> On Wed, 13 Mar 2013 10:00:10 -0400
> "Matthew M. DeLoera" <mdeloera-CERpylIU+L8AvxtiuMwx3w@public.gmane.org> wrote:
> 
>> Hello,
>> 
>> I've been searching to find a list of return codes from mount.cifs, with no luck. They're not listed in the manpage. I can google individual errors, but I'm hoping for something like a list of all possible errors. Or at least the most common ones.
>> 
>> Short of looking at code, is there any possibility that someone's documented details on something like causes for error code 5? Or other return values?
>> 
>> Any suggestions would be most appreciated!
>> 
>> Best regards,
>> - Matthew--
> 
> Like most mount helpers, the return codes are typically what the kernel
> returns on the mount() syscall. Error code '5' is likely EIO, which is
> a very generic error.
> 
> Cranking up debugging may give you more info as to what went wrong:
> 
>    https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting
> 
> -- 
> Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>