[2.6.39] CIFS write failures where 2.6.38 works

[2.6.39] CIFS write failures where 2.6.38 works

[Helge Hafting]

At work I use cifs for accessing a windows server. This has worked fine 
for a long time, up to and including Debian's 2.6.38-2.

I just installed Debians's 2.6.39-1, and had to give up on it.
Mounting CIFS works, and I can see the files. But if I
try to make a new file (with cp), I get a long delay.

In this time, anything accessing CIFS will pause, including "ls".
CIFS eventually becomes unstuck, and I will find that my "ls" succeeded, 
but the file copy did not. The file was created, but
with 0 size.

Trying to copy the file again results in another stall, and so on.


When I mount CIFS, I get this message:
CIFS VFS: Unexpected SMB signature

This happens with 2.6.38 as well as 2.6.39.

When writes fail in 2.6.39, I also get:
CIFS VFS: Send error in Close = -512

Followed by:
CIFS VFS: Server servername has not responded in 300 seconds. 
Reconnecting...

I have also seen
CIFS VFS: Send error in Close = -512
and
CIFS VFS: Send error in read = -13

Helge Hafting

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Suresh Jayaraman]

[Cc linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org]

On 06/01/2011 03:41 PM, Helge Hafting wrote:
> At work I use cifs for accessing a windows server. This has worked fine
> for a long time, up to and including Debian's 2.6.38-2.
> 
> I just installed Debians's 2.6.39-1, and had to give up on it.
> Mounting CIFS works, and I can see the files. But if I
> try to make a new file (with cp), I get a long delay.

What is the security mechanism you are using? If you seeing the problem
with ntlm, could you try using ntlmv2 and see whether the problem is
reproducible?


> In this time, anything accessing CIFS will pause, including "ls".
> CIFS eventually becomes unstuck, and I will find that my "ls" succeeded,
> but the file copy did not. The file was created, but
> with 0 size.
> 
> Trying to copy the file again results in another stall, and so on.
> 
> 
> When I mount CIFS, I get this message:
> CIFS VFS: Unexpected SMB signature
> 
> This happens with 2.6.38 as well as 2.6.39.
> 
> When writes fail in 2.6.39, I also get:
> CIFS VFS: Send error in Close = -512
> 
> Followed by:
> CIFS VFS: Server servername has not responded in 300 seconds.
> Reconnecting...
> 
> I have also seen
> CIFS VFS: Send error in Close = -512
> and
> CIFS VFS: Send error in read = -13

-- 
Suresh Jayaraman

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Suresh Jayaraman]

[Cc linux-cifs@vger.kernel.org]

On 06/01/2011 03:41 PM, Helge Hafting wrote:
> At work I use cifs for accessing a windows server. This has worked fine
> for a long time, up to and including Debian's 2.6.38-2.
> 
> I just installed Debians's 2.6.39-1, and had to give up on it.
> Mounting CIFS works, and I can see the files. But if I
> try to make a new file (with cp), I get a long delay.

What is the security mechanism you are using? If you seeing the problem
with ntlm, could you try using ntlmv2 and see whether the problem is
reproducible?


> In this time, anything accessing CIFS will pause, including "ls".
> CIFS eventually becomes unstuck, and I will find that my "ls" succeeded,
> but the file copy did not. The file was created, but
> with 0 size.
> 
> Trying to copy the file again results in another stall, and so on.
> 
> 
> When I mount CIFS, I get this message:
> CIFS VFS: Unexpected SMB signature
> 
> This happens with 2.6.38 as well as 2.6.39.
> 
> When writes fail in 2.6.39, I also get:
> CIFS VFS: Send error in Close = -512
> 
> Followed by:
> CIFS VFS: Server servername has not responded in 300 seconds.
> Reconnecting...
> 
> I have also seen
> CIFS VFS: Send error in Close = -512
> and
> CIFS VFS: Send error in read = -13

-- 
Suresh Jayaraman

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Fri, 03 Jun 2011 15:45:37 +0530
Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:

> [Cc linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org]
> 
> On 06/01/2011 03:41 PM, Helge Hafting wrote:
> > At work I use cifs for accessing a windows server. This has worked fine
> > for a long time, up to and including Debian's 2.6.38-2.
> > 
> > I just installed Debians's 2.6.39-1, and had to give up on it.
> > Mounting CIFS works, and I can see the files. But if I
> > try to make a new file (with cp), I get a long delay.
> 
> What is the security mechanism you are using? If you seeing the problem
> with ntlm, could you try using ntlmv2 and see whether the problem is
> reproducible?
> 
> 
> > In this time, anything accessing CIFS will pause, including "ls".
> > CIFS eventually becomes unstuck, and I will find that my "ls" succeeded,
> > but the file copy did not. The file was created, but
> > with 0 size.
> > 
> > Trying to copy the file again results in another stall, and so on.
> > 
> > 
> > When I mount CIFS, I get this message:
> > CIFS VFS: Unexpected SMB signature
> > 

I get this too on every mount using signing. It would be nice to fix
that but it's probably not related to the problem.

> > This happens with 2.6.38 as well as 2.6.39.
> > 
> > When writes fail in 2.6.39, I also get:
> > CIFS VFS: Send error in Close = -512
> > 

-512 is -ERESTARTSYS which probably means that it got a signal.

> > Followed by:
> > CIFS VFS: Server servername has not responded in 300 seconds.
> > Reconnecting...
> > 

So the server may not be responding to SMB echoes.

> > I have also seen
> > CIFS VFS: Send error in Close = -512
> > and
> > CIFS VFS: Send error in read = -13
> 

What sort of server is this? We'll probably need to see some wire
captures or debug logging to understand what's happening. It might be
good to open a bug at bugzilla.samba.org or bugzilla.kernel.org so we
can track progress on this.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Fri, 03 Jun 2011 15:45:37 +0530
Suresh Jayaraman <sjayaraman@suse.de> wrote:

> [Cc linux-cifs@vger.kernel.org]
> 
> On 06/01/2011 03:41 PM, Helge Hafting wrote:
> > At work I use cifs for accessing a windows server. This has worked fine
> > for a long time, up to and including Debian's 2.6.38-2.
> > 
> > I just installed Debians's 2.6.39-1, and had to give up on it.
> > Mounting CIFS works, and I can see the files. But if I
> > try to make a new file (with cp), I get a long delay.
> 
> What is the security mechanism you are using? If you seeing the problem
> with ntlm, could you try using ntlmv2 and see whether the problem is
> reproducible?
> 
> 
> > In this time, anything accessing CIFS will pause, including "ls".
> > CIFS eventually becomes unstuck, and I will find that my "ls" succeeded,
> > but the file copy did not. The file was created, but
> > with 0 size.
> > 
> > Trying to copy the file again results in another stall, and so on.
> > 
> > 
> > When I mount CIFS, I get this message:
> > CIFS VFS: Unexpected SMB signature
> > 

I get this too on every mount using signing. It would be nice to fix
that but it's probably not related to the problem.

> > This happens with 2.6.38 as well as 2.6.39.
> > 
> > When writes fail in 2.6.39, I also get:
> > CIFS VFS: Send error in Close = -512
> > 

-512 is -ERESTARTSYS which probably means that it got a signal.

> > Followed by:
> > CIFS VFS: Server servername has not responded in 300 seconds.
> > Reconnecting...
> > 

So the server may not be responding to SMB echoes.

> > I have also seen
> > CIFS VFS: Send error in Close = -512
> > and
> > CIFS VFS: Send error in read = -13
> 

What sort of server is this? We'll probably need to see some wire
captures or debug logging to understand what's happening. It might be
good to open a bug at bugzilla.samba.org or bugzilla.kernel.org so we
can track progress on this.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Helge Hafting]

On 03. juni 2011 12:15, Suresh Jayaraman wrote:
> [Cc linux-cifs@vger.kernel.org]
>
> On 06/01/2011 03:41 PM, Helge Hafting wrote:
>> At work I use cifs for accessing a windows server. This has worked fine
>> for a long time, up to and including Debian's 2.6.38-2.
>>
>> I just installed Debians's 2.6.39-1, and had to give up on it.
>> Mounting CIFS works, and I can see the files. But if I
>> try to make a new file (with cp), I get a long delay.
>
> What is the security mechanism you are using? If you seeing the problem
> with ntlm, could you try using ntlmv2 and see whether the problem is
> reproducible?

In the beginning, I did not specify the mechanism. So, whatever the 
default is.

The fstab entry was like this:
\\servername\resource /mountpoint cifs 
domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime 
0 0

I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It 
made no difference. Still failure with 2.6.39, and mounting with these 
new options works fine with 2.6.38

Helge Hafting

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Shirish Pargaonkar]

On Fri, Jun 3, 2011 at 10:11 AM, Helge Hafting <helge.hafting-CND4fxrytsY@public.gmane.org> wrote:
> On 03. juni 2011 12:15, Suresh Jayaraman wrote:
>>
>> [Cc linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org]
>>
>> On 06/01/2011 03:41 PM, Helge Hafting wrote:
>>>
>>> At work I use cifs for accessing a windows server. This has worked fine
>>> for a long time, up to and including Debian's 2.6.38-2.
>>>
>>> I just installed Debians's 2.6.39-1, and had to give up on it.
>>> Mounting CIFS works, and I can see the files. But if I
>>> try to make a new file (with cp), I get a long delay.
>>
>> What is the security mechanism you are using? If you seeing the problem
>> with ntlm, could you try using ntlmv2 and see whether the problem is
>> reproducible?
>
> In the beginning, I did not specify the mechanism. So, whatever the default
> is.
>
> The fstab entry was like this:
> \\servername\resource /mountpoint cifs
> domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime
> 0 0
>
> I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It made
> no difference. Still failure with 2.6.39, and mounting with these new
> options works fine with 2.6.38
>
> Helge Hafting
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Yes, I think wireshark trace would be really useful.

Regards,

Shirish

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Shirish Pargaonkar]

On Fri, Jun 3, 2011 at 10:11 AM, Helge Hafting <helge.hafting@hist.no> wrote:
> On 03. juni 2011 12:15, Suresh Jayaraman wrote:
>>
>> [Cc linux-cifs@vger.kernel.org]
>>
>> On 06/01/2011 03:41 PM, Helge Hafting wrote:
>>>
>>> At work I use cifs for accessing a windows server. This has worked fine
>>> for a long time, up to and including Debian's 2.6.38-2.
>>>
>>> I just installed Debians's 2.6.39-1, and had to give up on it.
>>> Mounting CIFS works, and I can see the files. But if I
>>> try to make a new file (with cp), I get a long delay.
>>
>> What is the security mechanism you are using? If you seeing the problem
>> with ntlm, could you try using ntlmv2 and see whether the problem is
>> reproducible?
>
> In the beginning, I did not specify the mechanism. So, whatever the default
> is.
>
> The fstab entry was like this:
> \\servername\resource /mountpoint cifs
> domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime
> 0 0
>
> I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It made
> no difference. Still failure with 2.6.39, and mounting with these new
> options works fine with 2.6.38
>
> Helge Hafting
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Yes, I think wireshark trace would be really useful.

Regards,

Shirish

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Fri, 03 Jun 2011 17:11:05 +0200
Helge Hafting <helge.hafting-CND4fxrytsY@public.gmane.org> wrote:

> On 03. juni 2011 12:15, Suresh Jayaraman wrote:
> > [Cc linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org]
> >
> > On 06/01/2011 03:41 PM, Helge Hafting wrote:
> >> At work I use cifs for accessing a windows server. This has worked fine
> >> for a long time, up to and including Debian's 2.6.38-2.
> >>
> >> I just installed Debians's 2.6.39-1, and had to give up on it.
> >> Mounting CIFS works, and I can see the files. But if I
> >> try to make a new file (with cp), I get a long delay.
> >
> > What is the security mechanism you are using? If you seeing the problem
> > with ntlm, could you try using ntlmv2 and see whether the problem is
> > reproducible?
> 
> In the beginning, I did not specify the mechanism. So, whatever the 
> default is.
> 
> The fstab entry was like this:
> \\servername\resource /mountpoint cifs 
> domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime 
> 0 0
> 
> I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It 
> made no difference. Still failure with 2.6.39, and mounting with these 
> new options works fine with 2.6.38
> 

I think we need to understand what's happening on the wire. Are you
still able to reproduce this? If so, can you turn up debug logging and
reproduce this?. Instructions for how to do that are here:

    http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Enabling_Debugging

Also, it looks like someone opened a bug at kernel.org too:

    https://bugzilla.kernel.org/show_bug.cgi?id=36952

...so if you can attach the resulting log there, that would be great.

Thanks,
-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Fri, 03 Jun 2011 17:11:05 +0200
Helge Hafting <helge.hafting@hist.no> wrote:

> On 03. juni 2011 12:15, Suresh Jayaraman wrote:
> > [Cc linux-cifs@vger.kernel.org]
> >
> > On 06/01/2011 03:41 PM, Helge Hafting wrote:
> >> At work I use cifs for accessing a windows server. This has worked fine
> >> for a long time, up to and including Debian's 2.6.38-2.
> >>
> >> I just installed Debians's 2.6.39-1, and had to give up on it.
> >> Mounting CIFS works, and I can see the files. But if I
> >> try to make a new file (with cp), I get a long delay.
> >
> > What is the security mechanism you are using? If you seeing the problem
> > with ntlm, could you try using ntlmv2 and see whether the problem is
> > reproducible?
> 
> In the beginning, I did not specify the mechanism. So, whatever the 
> default is.
> 
> The fstab entry was like this:
> \\servername\resource /mountpoint cifs 
> domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime 
> 0 0
> 
> I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It 
> made no difference. Still failure with 2.6.39, and mounting with these 
> new options works fine with 2.6.38
> 

I think we need to understand what's happening on the wire. Are you
still able to reproduce this? If so, can you turn up debug logging and
reproduce this?. Instructions for how to do that are here:

    http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Enabling_Debugging

Also, it looks like someone opened a bug at kernel.org too:

    https://bugzilla.kernel.org/show_bug.cgi?id=36952

...so if you can attach the resulting log there, that would be great.

Thanks,
-- 
Jeff Layton <jlayton@redhat.com>

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Thu, 9 Jun 2011 18:28:45 -0400
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Fri, 03 Jun 2011 17:11:05 +0200
> Helge Hafting <helge.hafting-CND4fxrytsY@public.gmane.org> wrote:
> 
> > On 03. juni 2011 12:15, Suresh Jayaraman wrote:
> > > [Cc linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org]
> > >
> > > On 06/01/2011 03:41 PM, Helge Hafting wrote:
> > >> At work I use cifs for accessing a windows server. This has worked fine
> > >> for a long time, up to and including Debian's 2.6.38-2.
> > >>
> > >> I just installed Debians's 2.6.39-1, and had to give up on it.
> > >> Mounting CIFS works, and I can see the files. But if I
> > >> try to make a new file (with cp), I get a long delay.
> > >
> > > What is the security mechanism you are using? If you seeing the problem
> > > with ntlm, could you try using ntlmv2 and see whether the problem is
> > > reproducible?
> > 
> > In the beginning, I did not specify the mechanism. So, whatever the 
> > default is.
> > 
> > The fstab entry was like this:
> > \\servername\resource /mountpoint cifs 
> > domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime 
> > 0 0
> > 
> > I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It 
> > made no difference. Still failure with 2.6.39, and mounting with these 
> > new options works fine with 2.6.38
> > 
> 
> I think we need to understand what's happening on the wire. Are you
> still able to reproduce this? If so, can you turn up debug logging and
> reproduce this?. Instructions for how to do that are here:
> 
>     http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Enabling_Debugging
> 
> Also, it looks like someone opened a bug at kernel.org too:
> 
>     https://bugzilla.kernel.org/show_bug.cgi?id=36952
> 
> ...so if you can attach the resulting log there, that would be great.
> 

I think that this is probably due to the change that added the
page_mkwrite function to cifs.ko. Prior to that, cifs did single-page
writes on signed connections. Now we do multi-page writes and windows
servers apparently reject large write calls on signed connections.

One way to test this theory would be to set the wsize to something
smaller when you mount. For instance:

    wsize=16384

...assuming that doesn't go over the server's MaxBufferSize, then that
should act as a workaround. Can you try that and let me know if it
helps?

Thanks,
-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Thu, 9 Jun 2011 18:28:45 -0400
Jeff Layton <jlayton@redhat.com> wrote:

> On Fri, 03 Jun 2011 17:11:05 +0200
> Helge Hafting <helge.hafting@hist.no> wrote:
> 
> > On 03. juni 2011 12:15, Suresh Jayaraman wrote:
> > > [Cc linux-cifs@vger.kernel.org]
> > >
> > > On 06/01/2011 03:41 PM, Helge Hafting wrote:
> > >> At work I use cifs for accessing a windows server. This has worked fine
> > >> for a long time, up to and including Debian's 2.6.38-2.
> > >>
> > >> I just installed Debians's 2.6.39-1, and had to give up on it.
> > >> Mounting CIFS works, and I can see the files. But if I
> > >> try to make a new file (with cp), I get a long delay.
> > >
> > > What is the security mechanism you are using? If you seeing the problem
> > > with ntlm, could you try using ntlmv2 and see whether the problem is
> > > reproducible?
> > 
> > In the beginning, I did not specify the mechanism. So, whatever the 
> > default is.
> > 
> > The fstab entry was like this:
> > \\servername\resource /mountpoint cifs 
> > domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime 
> > 0 0
> > 
> > I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It 
> > made no difference. Still failure with 2.6.39, and mounting with these 
> > new options works fine with 2.6.38
> > 
> 
> I think we need to understand what's happening on the wire. Are you
> still able to reproduce this? If so, can you turn up debug logging and
> reproduce this?. Instructions for how to do that are here:
> 
>     http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Enabling_Debugging
> 
> Also, it looks like someone opened a bug at kernel.org too:
> 
>     https://bugzilla.kernel.org/show_bug.cgi?id=36952
> 
> ...so if you can attach the resulting log there, that would be great.
> 

I think that this is probably due to the change that added the
page_mkwrite function to cifs.ko. Prior to that, cifs did single-page
writes on signed connections. Now we do multi-page writes and windows
servers apparently reject large write calls on signed connections.

One way to test this theory would be to set the wsize to something
smaller when you mount. For instance:

    wsize=16384

...assuming that doesn't go over the server's MaxBufferSize, then that
should act as a workaround. Can you try that and let me know if it
helps?

Thanks,
-- 
Jeff Layton <jlayton@redhat.com>

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Helge Hafting]

On 22. juni 2011 22:36, Jeff Layton wrote:
> On Thu, 9 Jun 2011 18:28:45 -0400
> Jeff Layton<jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>  wrote:
>
>> On Fri, 03 Jun 2011 17:11:05 +0200
>> Helge Hafting<helge.hafting-CND4fxrytsY@public.gmane.org>  wrote:
>>
>>> On 03. juni 2011 12:15, Suresh Jayaraman wrote:
>>>> [Cc linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org]
>>>>
>>>> On 06/01/2011 03:41 PM, Helge Hafting wrote:
>>>>> At work I use cifs for accessing a windows server. This has worked fine
>>>>> for a long time, up to and including Debian's 2.6.38-2.
>>>>>
>>>>> I just installed Debians's 2.6.39-1, and had to give up on it.
>>>>> Mounting CIFS works, and I can see the files. But if I
>>>>> try to make a new file (with cp), I get a long delay.
>>>>
>>>> What is the security mechanism you are using? If you seeing the problem
>>>> with ntlm, could you try using ntlmv2 and see whether the problem is
>>>> reproducible?
>>>
>>> In the beginning, I did not specify the mechanism. So, whatever the
>>> default is.
>>>
>>> The fstab entry was like this:
>>> \\servername\resource /mountpoint cifs
>>> domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime
>>> 0 0
>>>
>>> I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It
>>> made no difference. Still failure with 2.6.39, and mounting with these
>>> new options works fine with 2.6.38
>>>
>>
>> I think we need to understand what's happening on the wire. Are you
>> still able to reproduce this? If so, can you turn up debug logging and
>> reproduce this?. Instructions for how to do that are here:
>>
>>      http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Enabling_Debugging
>>
>> Also, it looks like someone opened a bug at kernel.org too:
>>
>>      https://bugzilla.kernel.org/show_bug.cgi?id=36952
>>
>> ...so if you can attach the resulting log there, that would be great.
>>
>
> I think that this is probably due to the change that added the
> page_mkwrite function to cifs.ko. Prior to that, cifs did single-page
> writes on signed connections. Now we do multi-page writes and windows
> servers apparently reject large write calls on signed connections.
>
> One way to test this theory would be to set the wsize to something
> smaller when you mount. For instance:
>
>      wsize=16384
>
> ...assuming that doesn't go over the server's MaxBufferSize, then that
> should act as a workaround. Can you try that and let me know if it
> helps?


Yes, that seemed to fix it. I added wsize=16384 and mounted using 
debians 2.6.39-1-amd64 kernel.

I tried a recursive copy of 26MB from one directory tree to another on 
that mount. It completed in 24s with no error messages. 1MB/s is not 
much, but there may be 40 other users.

The server runs windows 2008r2, 64-bit.

Helge Hafting

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Helge Hafting]

On 22. juni 2011 22:36, Jeff Layton wrote:
> On Thu, 9 Jun 2011 18:28:45 -0400
> Jeff Layton<jlayton@redhat.com>  wrote:
>
>> On Fri, 03 Jun 2011 17:11:05 +0200
>> Helge Hafting<helge.hafting@hist.no>  wrote:
>>
>>> On 03. juni 2011 12:15, Suresh Jayaraman wrote:
>>>> [Cc linux-cifs@vger.kernel.org]
>>>>
>>>> On 06/01/2011 03:41 PM, Helge Hafting wrote:
>>>>> At work I use cifs for accessing a windows server. This has worked fine
>>>>> for a long time, up to and including Debian's 2.6.38-2.
>>>>>
>>>>> I just installed Debians's 2.6.39-1, and had to give up on it.
>>>>> Mounting CIFS works, and I can see the files. But if I
>>>>> try to make a new file (with cp), I get a long delay.
>>>>
>>>> What is the security mechanism you are using? If you seeing the problem
>>>> with ntlm, could you try using ntlmv2 and see whether the problem is
>>>> reproducible?
>>>
>>> In the beginning, I did not specify the mechanism. So, whatever the
>>> default is.
>>>
>>> The fstab entry was like this:
>>> \\servername\resource /mountpoint cifs
>>> domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime
>>> 0 0
>>>
>>> I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It
>>> made no difference. Still failure with 2.6.39, and mounting with these
>>> new options works fine with 2.6.38
>>>
>>
>> I think we need to understand what's happening on the wire. Are you
>> still able to reproduce this? If so, can you turn up debug logging and
>> reproduce this?. Instructions for how to do that are here:
>>
>>      http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Enabling_Debugging
>>
>> Also, it looks like someone opened a bug at kernel.org too:
>>
>>      https://bugzilla.kernel.org/show_bug.cgi?id=36952
>>
>> ...so if you can attach the resulting log there, that would be great.
>>
>
> I think that this is probably due to the change that added the
> page_mkwrite function to cifs.ko. Prior to that, cifs did single-page
> writes on signed connections. Now we do multi-page writes and windows
> servers apparently reject large write calls on signed connections.
>
> One way to test this theory would be to set the wsize to something
> smaller when you mount. For instance:
>
>      wsize=16384
>
> ...assuming that doesn't go over the server's MaxBufferSize, then that
> should act as a workaround. Can you try that and let me know if it
> helps?


Yes, that seemed to fix it. I added wsize=16384 and mounted using 
debians 2.6.39-1-amd64 kernel.

I tried a recursive copy of 26MB from one directory tree to another on 
that mount. It completed in 24s with no error messages. 1MB/s is not 
much, but there may be 40 other users.

The server runs windows 2008r2, 64-bit.

Helge Hafting

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Thu, 23 Jun 2011 13:25:36 +0200
Helge Hafting <helge.hafting@hist.no> wrote:

> On 22. juni 2011 22:36, Jeff Layton wrote:
> > On Thu, 9 Jun 2011 18:28:45 -0400
> > Jeff Layton<jlayton@redhat.com>  wrote:
> >
> >> On Fri, 03 Jun 2011 17:11:05 +0200
> >> Helge Hafting<helge.hafting@hist.no>  wrote:
> >>
> >>> On 03. juni 2011 12:15, Suresh Jayaraman wrote:
> >>>> [Cc linux-cifs@vger.kernel.org]
> >>>>
> >>>> On 06/01/2011 03:41 PM, Helge Hafting wrote:
> >>>>> At work I use cifs for accessing a windows server. This has worked fine
> >>>>> for a long time, up to and including Debian's 2.6.38-2.
> >>>>>
> >>>>> I just installed Debians's 2.6.39-1, and had to give up on it.
> >>>>> Mounting CIFS works, and I can see the files. But if I
> >>>>> try to make a new file (with cp), I get a long delay.
> >>>>
> >>>> What is the security mechanism you are using? If you seeing the problem
> >>>> with ntlm, could you try using ntlmv2 and see whether the problem is
> >>>> reproducible?
> >>>
> >>> In the beginning, I did not specify the mechanism. So, whatever the
> >>> default is.
> >>>
> >>> The fstab entry was like this:
> >>> \\servername\resource /mountpoint cifs
> >>> domain=MYDOMAIN,credentials=/etc/fstabcred,rw,noauto,iocharset=utf8,uid=username,gid=group,sockopt=TCP_NODELAY,users,file_mode=0640,dir_mode=0750,relatime
> >>> 0 0
> >>>
> >>> I looked at cifs options, and tried to add "sign" and "sec=ntlmv2i". It
> >>> made no difference. Still failure with 2.6.39, and mounting with these
> >>> new options works fine with 2.6.38
> >>>
> >>
> >> I think we need to understand what's happening on the wire. Are you
> >> still able to reproduce this? If so, can you turn up debug logging and
> >> reproduce this?. Instructions for how to do that are here:
> >>
> >>      http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Enabling_Debugging
> >>
> >> Also, it looks like someone opened a bug at kernel.org too:
> >>
> >>      https://bugzilla.kernel.org/show_bug.cgi?id=36952
> >>
> >> ...so if you can attach the resulting log there, that would be great.
> >>
> >
> > I think that this is probably due to the change that added the
> > page_mkwrite function to cifs.ko. Prior to that, cifs did single-page
> > writes on signed connections. Now we do multi-page writes and windows
> > servers apparently reject large write calls on signed connections.
> >
> > One way to test this theory would be to set the wsize to something
> > smaller when you mount. For instance:
> >
> >      wsize=16384
> >
> > ...assuming that doesn't go over the server's MaxBufferSize, then that
> > should act as a workaround. Can you try that and let me know if it
> > helps?
> 
> 
> Yes, that seemed to fix it. I added wsize=16384 and mounted using 
> debians 2.6.39-1-amd64 kernel.
> 
> I tried a recursive copy of 26MB from one directory tree to another on 
> that mount. It completed in 24s with no error messages. 1MB/s is not 
> much, but there may be 40 other users.
> 
> The server runs windows 2008r2, 64-bit.
> 

Great, thanks for testing that. I've already sent Steve French a patch
that should fix this in 3.0, but we'll need to do something different
for 2.6.39 stable. I'll roll up a patch for that soon.

BTW, 3.0 should have much improved write performance, even with signing
enabled as that adds the capability to do async writes.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Thu, 23 Jun 2011 13:25:36 +0200
Helge Hafting <helge.hafting-CND4fxrytsY@public.gmane.org> wrote:

> >
> > I think that this is probably due to the change that added the
> > page_mkwrite function to cifs.ko. Prior to that, cifs did single-page
> > writes on signed connections. Now we do multi-page writes and windows
> > servers apparently reject large write calls on signed connections.
> >
> > One way to test this theory would be to set the wsize to something
> > smaller when you mount. For instance:
> >
> >      wsize=16384
> >
> > ...assuming that doesn't go over the server's MaxBufferSize, then thatFrom 
> > should act as a workaround. Can you try that and let me know if it
> > helps?
> 
> 
> Yes, that seemed to fix it. I added wsize=16384 and mounted using 
> debians 2.6.39-1-amd64 kernel.
> 
> I tried a recursive copy of 26MB from one directory tree to another on 
> that mount. It completed in 24s with no error messages. 1MB/s is not 
> much, but there may be 40 other users.
> 
> The server runs windows 2008r2, 64-bit.
> 

Great, thanks for testing that. Would you be able to test this patch on
2.6.39 kernel? It should apply cleanly there, but let me know if it
doesn't.

Also, I'm planning to send this off to stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org if there are no
objections, so if anyone has any, let me know ASAP.

--------------------[snip]-----------------------

cifs: fix wsize negotiation for 2.6.39 stable

Prior to 2.6.39, when signing was enabled on a socket the client only
sent single-page writes. This changed with commit ca83ce3d5b, which
made signed and unsigned connections use the same codepaths for write
calls.

This caused a regression when working with windows servers. Windows
machines will reject writes larger than the MaxBufferSize when signing
is active, but does not clear the CAP_LARGE_WRITE_X flag in the
negotiation.

This patch backports 2 patches that fix this problem in 3.0 kernels,
and changes a couple of the constants to values appropriate for the
writeback code in 2.6.39.

Cc: stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Reported-by: Helge Hafting <helge.hafting-CND4fxrytsY@public.gmane.org>
Signed-off-by: Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 fs/cifs/connect.c |   64 ++++++++++++++++++++++++++++++++++++----------------
 1 files changed, 44 insertions(+), 20 deletions(-)

diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index 29fac128..2a9d9c1 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -2571,23 +2571,6 @@ static void setup_cifs_sb(struct smb_vol *pvolume_info,
 	else /* default */
 		cifs_sb->rsize = CIFSMaxBufSize;
 
-	if (pvolume_info->wsize > PAGEVEC_SIZE * PAGE_CACHE_SIZE) {
-		cERROR(1, "wsize %d too large, using 4096 instead",
-			  pvolume_info->wsize);
-		cifs_sb->wsize = 4096;
-	} else if (pvolume_info->wsize)
-		cifs_sb->wsize = pvolume_info->wsize;
-	else
-		cifs_sb->wsize = min_t(const int,
-					PAGEVEC_SIZE * PAGE_CACHE_SIZE,
-					127*1024);
-		/* old default of CIFSMaxBufSize was too small now
-		   that SMB Write2 can send multiple pages in kvec.
-		   RFC1001 does not describe what happens when frame
-		   bigger than 128K is sent so use that as max in
-		   conjunction with 52K kvec constraint on arch with 4K
-		   page size  */
-
 	if (cifs_sb->rsize < 2048) {
 		cifs_sb->rsize = 2048;
 		/* Windows ME may prefer this */
@@ -2665,6 +2648,48 @@ static void setup_cifs_sb(struct smb_vol *pvolume_info,
 			   "mount option supported");
 }
 
+/* Prior to 3.0, cifs couldn't handle writes larger than this */
+#define CIFS_MAX_WSIZE (PAGEVEC_SIZE * PAGE_CACHE_SIZE)
+
+/*
+ * When the server doesn't allow large posix writes, only allow a wsize of
+ * 128k minus the size of the WRITE_AND_X header. That allows for a write up
+ * to the maximum size described by RFC1002.
+ */
+#define CIFS_MAX_RFC1002_WSIZE (128 * 1024 - sizeof(WRITE_REQ) + 4)
+
+/* Make the default the same as the max */
+#define CIFS_DEFAULT_WSIZE CIFS_MAX_WSIZE
+
+static unsigned int
+cifs_negotiate_wsize(struct cifsTconInfo *tcon, struct smb_vol *pvolume_info)
+{
+	__u64 unix_cap = le64_to_cpu(tcon->fsUnixInfo.Capability);
+	struct TCP_Server_Info *server = tcon->ses->server;
+	unsigned int wsize = pvolume_info->wsize ? pvolume_info->wsize :
+				CIFS_DEFAULT_WSIZE;
+
+	/* can server support 24-bit write sizes? (via UNIX extensions) */
+	if (!tcon->unix_ext || !(unix_cap & CIFS_UNIX_LARGE_WRITE_CAP))
+		wsize = min_t(unsigned int, wsize, CIFS_MAX_RFC1002_WSIZE);
+
+	/*
+	 * no CAP_LARGE_WRITE_X or is signing enabled without CAP_UNIX set?
+	 * Limit it to max buffer offered by the server, minus the size of the
+	 * WRITEX header, not including the 4 byte RFC1001 length.
+	 */
+	if (!(server->capabilities & CAP_LARGE_WRITE_X) ||
+	    (!(server->capabilities & CAP_UNIX) &&
+	     (server->sec_mode & (SECMODE_SIGN_ENABLED|SECMODE_SIGN_REQUIRED))))
+		wsize = min_t(unsigned int, wsize,
+				server->maxBuf - sizeof(WRITE_REQ) + 4);
+
+	/* hard limit of CIFS_MAX_WSIZE */
+	wsize = min_t(unsigned int, wsize, CIFS_MAX_WSIZE);
+
+	return wsize;
+}
+
 static int
 is_path_accessible(int xid, struct cifsTconInfo *tcon,
 		   struct cifs_sb_info *cifs_sb, const char *full_path)
@@ -2866,13 +2891,12 @@ try_mount_again:
 		cifs_sb->rsize = 1024 * 127;
 		cFYI(DBG2, "no very large read support, rsize now 127K");
 	}
-	if (!(tcon->ses->capabilities & CAP_LARGE_WRITE_X))
-		cifs_sb->wsize = min(cifs_sb->wsize,
-			       (tcon->ses->server->maxBuf - MAX_CIFS_HDR_SIZE));
 	if (!(tcon->ses->capabilities & CAP_LARGE_READ_X))
 		cifs_sb->rsize = min(cifs_sb->rsize,
 			       (tcon->ses->server->maxBuf - MAX_CIFS_HDR_SIZE));
 
+	cifs_sb->wsize = cifs_negotiate_wsize(tcon, volume_info);
+
 remote_path_check:
 	/* check if a whole path (including prepath) is not remote */
 	if (!rc && tcon) {
-- 
1.7.5.4

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Jeff Layton]

On Thu, 23 Jun 2011 13:25:36 +0200
Helge Hafting <helge.hafting@hist.no> wrote:

> >
> > I think that this is probably due to the change that added the
> > page_mkwrite function to cifs.ko. Prior to that, cifs did single-page
> > writes on signed connections. Now we do multi-page writes and windows
> > servers apparently reject large write calls on signed connections.
> >
> > One way to test this theory would be to set the wsize to something
> > smaller when you mount. For instance:
> >
> >      wsize=16384
> >
> > ...assuming that doesn't go over the server's MaxBufferSize, then thatFrom 
> > should act as a workaround. Can you try that and let me know if it
> > helps?
> 
> 
> Yes, that seemed to fix it. I added wsize=16384 and mounted using 
> debians 2.6.39-1-amd64 kernel.
> 
> I tried a recursive copy of 26MB from one directory tree to another on 
> that mount. It completed in 24s with no error messages. 1MB/s is not 
> much, but there may be 40 other users.
> 
> The server runs windows 2008r2, 64-bit.
> 

Great, thanks for testing that. Would you be able to test this patch on
2.6.39 kernel? It should apply cleanly there, but let me know if it
doesn't.

Also, I'm planning to send this off to stable@kernel.org if there are no
objections, so if anyone has any, let me know ASAP.

--------------------[snip]-----------------------

cifs: fix wsize negotiation for 2.6.39 stable

Prior to 2.6.39, when signing was enabled on a socket the client only
sent single-page writes. This changed with commit ca83ce3d5b, which
made signed and unsigned connections use the same codepaths for write
calls.

This caused a regression when working with windows servers. Windows
machines will reject writes larger than the MaxBufferSize when signing
is active, but does not clear the CAP_LARGE_WRITE_X flag in the
negotiation.

This patch backports 2 patches that fix this problem in 3.0 kernels,
and changes a couple of the constants to values appropriate for the
writeback code in 2.6.39.

Cc: stable@kernel.org
Reported-by: Helge Hafting <helge.hafting@hist.no>
Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 fs/cifs/connect.c |   64 ++++++++++++++++++++++++++++++++++++----------------
 1 files changed, 44 insertions(+), 20 deletions(-)

diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index 29fac128..2a9d9c1 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -2571,23 +2571,6 @@ static void setup_cifs_sb(struct smb_vol *pvolume_info,
 	else /* default */
 		cifs_sb->rsize = CIFSMaxBufSize;
 
-	if (pvolume_info->wsize > PAGEVEC_SIZE * PAGE_CACHE_SIZE) {
-		cERROR(1, "wsize %d too large, using 4096 instead",
-			  pvolume_info->wsize);
-		cifs_sb->wsize = 4096;
-	} else if (pvolume_info->wsize)
-		cifs_sb->wsize = pvolume_info->wsize;
-	else
-		cifs_sb->wsize = min_t(const int,
-					PAGEVEC_SIZE * PAGE_CACHE_SIZE,
-					127*1024);
-		/* old default of CIFSMaxBufSize was too small now
-		   that SMB Write2 can send multiple pages in kvec.
-		   RFC1001 does not describe what happens when frame
-		   bigger than 128K is sent so use that as max in
-		   conjunction with 52K kvec constraint on arch with 4K
-		   page size  */
-
 	if (cifs_sb->rsize < 2048) {
 		cifs_sb->rsize = 2048;
 		/* Windows ME may prefer this */
@@ -2665,6 +2648,48 @@ static void setup_cifs_sb(struct smb_vol *pvolume_info,
 			   "mount option supported");
 }
 
+/* Prior to 3.0, cifs couldn't handle writes larger than this */
+#define CIFS_MAX_WSIZE (PAGEVEC_SIZE * PAGE_CACHE_SIZE)
+
+/*
+ * When the server doesn't allow large posix writes, only allow a wsize of
+ * 128k minus the size of the WRITE_AND_X header. That allows for a write up
+ * to the maximum size described by RFC1002.
+ */
+#define CIFS_MAX_RFC1002_WSIZE (128 * 1024 - sizeof(WRITE_REQ) + 4)
+
+/* Make the default the same as the max */
+#define CIFS_DEFAULT_WSIZE CIFS_MAX_WSIZE
+
+static unsigned int
+cifs_negotiate_wsize(struct cifsTconInfo *tcon, struct smb_vol *pvolume_info)
+{
+	__u64 unix_cap = le64_to_cpu(tcon->fsUnixInfo.Capability);
+	struct TCP_Server_Info *server = tcon->ses->server;
+	unsigned int wsize = pvolume_info->wsize ? pvolume_info->wsize :
+				CIFS_DEFAULT_WSIZE;
+
+	/* can server support 24-bit write sizes? (via UNIX extensions) */
+	if (!tcon->unix_ext || !(unix_cap & CIFS_UNIX_LARGE_WRITE_CAP))
+		wsize = min_t(unsigned int, wsize, CIFS_MAX_RFC1002_WSIZE);
+
+	/*
+	 * no CAP_LARGE_WRITE_X or is signing enabled without CAP_UNIX set?
+	 * Limit it to max buffer offered by the server, minus the size of the
+	 * WRITEX header, not including the 4 byte RFC1001 length.
+	 */
+	if (!(server->capabilities & CAP_LARGE_WRITE_X) ||
+	    (!(server->capabilities & CAP_UNIX) &&
+	     (server->sec_mode & (SECMODE_SIGN_ENABLED|SECMODE_SIGN_REQUIRED))))
+		wsize = min_t(unsigned int, wsize,
+				server->maxBuf - sizeof(WRITE_REQ) + 4);
+
+	/* hard limit of CIFS_MAX_WSIZE */
+	wsize = min_t(unsigned int, wsize, CIFS_MAX_WSIZE);
+
+	return wsize;
+}
+
 static int
 is_path_accessible(int xid, struct cifsTconInfo *tcon,
 		   struct cifs_sb_info *cifs_sb, const char *full_path)
@@ -2866,13 +2891,12 @@ try_mount_again:
 		cifs_sb->rsize = 1024 * 127;
 		cFYI(DBG2, "no very large read support, rsize now 127K");
 	}
-	if (!(tcon->ses->capabilities & CAP_LARGE_WRITE_X))
-		cifs_sb->wsize = min(cifs_sb->wsize,
-			       (tcon->ses->server->maxBuf - MAX_CIFS_HDR_SIZE));
 	if (!(tcon->ses->capabilities & CAP_LARGE_READ_X))
 		cifs_sb->rsize = min(cifs_sb->rsize,
 			       (tcon->ses->server->maxBuf - MAX_CIFS_HDR_SIZE));
 
+	cifs_sb->wsize = cifs_negotiate_wsize(tcon, volume_info);
+
 remote_path_check:
 	/* check if a whole path (including prepath) is not remote */
 	if (!rc && tcon) {
-- 
1.7.5.4

Re: [2.6.39] CIFS write failures where 2.6.38 works

[Maciej Rutecki]

I created a Bugzilla entry at 
https://bugzilla.kernel.org/show_bug.cgi?id=36952
for your bug report, please add your address to the CC list in there, thanks!

On środa, 1 czerwca 2011 o 12:11:10 Helge Hafting wrote:
> At work I use cifs for accessing a windows server. This has worked fine
> for a long time, up to and including Debian's 2.6.38-2.
> 
> I just installed Debians's 2.6.39-1, and had to give up on it.
> Mounting CIFS works, and I can see the files. But if I
> try to make a new file (with cp), I get a long delay.
> 
> In this time, anything accessing CIFS will pause, including "ls".
> CIFS eventually becomes unstuck, and I will find that my "ls" succeeded,
> but the file copy did not. The file was created, but
> with 0 size.
> 
> Trying to copy the file again results in another stall, and so on.
> 
> 
> When I mount CIFS, I get this message:
> CIFS VFS: Unexpected SMB signature
> 
> This happens with 2.6.38 as well as 2.6.39.
> 
> When writes fail in 2.6.39, I also get:
> CIFS VFS: Send error in Close = -512
> 
> Followed by:
> CIFS VFS: Server servername has not responded in 300 seconds.
> Reconnecting...
> 
> I have also seen
> CIFS VFS: Send error in Close = -512
> and
> CIFS VFS: Send error in read = -13
> 
> Helge Hafting
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Maciej Rutecki
http://www.maciek.unixy.pl