linux-cifs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Locking issue between NFSv4 and SMB client
@ 2021-07-15 23:45 dai.ngo
  2021-07-15 23:47 ` dai.ngo
  2021-09-23 21:50 ` Bruce Fields
  0 siblings, 2 replies; 15+ messages in thread
From: dai.ngo @ 2021-07-15 23:45 UTC (permalink / raw)
  To: Bruce Fields; +Cc: linux-nfs, CIFS

Hi Bruce,

I'm doing some locking testing between NFSv4 and SMB client and
think there are some issues on the server that allows both clients
to lock the same file at the same time.

Here is what I did:

NOTE: lck is a simple program that use lockf(3) to lock a file from
offset 0 to the length specified by '-l'.

On NFSv4 client
---------------

[root@nfsvmd07 ~]# nfsstat -m
/tmp/mnt from nfsvmf24:/root/smb_share
Flags:	rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
        proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
        local_lock=none,addr=10.80.111.94
[root@nfsvmd07 ~]#


[root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
Lck/file: 1, Maxlocks: 10000000
Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
LOCKED...

Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop

[NFS client successfully locks the file]

On SMB client
-------------

[root@nfsvme24 ~]# mount |grep cifs
//nfsvmf24/smb_share on /tmp/mnt type cifs (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
[root@nfsvme24 ~]#

[root@nfsvme24 ~]# smbclient -L nfsvmf24
Enter SAMBA\root's password:

	Sharename       Type      Comment
	---------       ----      -------
	print$          Disk      Printer Drivers
	smb_share       Disk      Test Samba Share       <<===== share to mount
	IPC$            IPC       IPC Service (Samba 4.10.16)
	root            Disk      Home Directories
Reconnecting with SMB1 for workgroup listing.

	Server               Comment
	---------            -------

	Workgroup            Master
	---------            -------
[root@nfsvme24 ~]#

[root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
Lck/file: 1, Maxlocks: 10000000
Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
LOCKED...

Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop

[SMB client successfully locks the file]

The same issue happens when either client locks the file first.
I think this is what has happened:

1. NFSv4 client opens and locks the file first

     . NFSv4 client send OPEN and LOCK to server, server replies
       OK on both requests.

     . SMB client sends create request with Oplock==Lease for
       the same file.

     . server holds off on replying to SMB client's create request,
       recalls delegation from NFSv4 client, waits for NFSv4 client
       to return the delegation then replies success to SMB client's
       create request with lease granted (Oplock==Lease).

       NOTE: I think SMB server should replies the create request
       with Oplock==None to force the SMB client to sends the
       lock request.

     . Once SMB client receives the reply of the create with
       'Oplock==Lease', it assumes it has full control of the file
       therefor it does not need to send the lock request.

     . both NFSv4 and SMB client now think they have locked the file.

pcap:  nfs_lock_smb_lock.pcap

2. SMB client creates the file with 'Oplock==Lease' first

     . SMB sends create request with 'Oplock==Lease' to server,
       server replies OK with 'Oplock==Lease'. SMB client skips
       sending lock request since it assumes it has full control
       of the file with the lease.

     . NFSv4 client sends OPEN to server, server replies OK with
       delagation is none. NFSv4 client sends LOCK request, since
       no lock was created in the kernel for the SMB client, the
       lock was granted to the NFSv4 client.

      NOTE: I think the SMB server should send lease break
      notification to the SMB client, wait for the lease break
      acknowledgment from SMB client before replying to the
      OPEN of the NFSv4 client. This will force the SMB client
      to send the lock request to the server.

     . both NFSv4 and SMB client now think they have locked the file.

Your thought?

Thanks,

-Dai


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-07-15 23:45 Locking issue between NFSv4 and SMB client dai.ngo
@ 2021-07-15 23:47 ` dai.ngo
  2021-09-23 21:50 ` Bruce Fields
  1 sibling, 0 replies; 15+ messages in thread
From: dai.ngo @ 2021-07-15 23:47 UTC (permalink / raw)
  To: Bruce Fields; +Cc: linux-nfs, CIFS

[-- Attachment #1: Type: text/plain, Size: 4649 bytes --]

Oops I forgot the pcap files

-Dai

On 7/15/21 4:45 PM, dai.ngo@oracle.com wrote:
> Hi Bruce,
>
> I'm doing some locking testing between NFSv4 and SMB client and
> think there are some issues on the server that allows both clients
> to lock the same file at the same time.
>
> Here is what I did:
>
> NOTE: lck is a simple program that use lockf(3) to lock a file from
> offset 0 to the length specified by '-l'.
>
> On NFSv4 client
> ---------------
>
> [root@nfsvmd07 ~]# nfsstat -m
> /tmp/mnt from nfsvmf24:/root/smb_share
> Flags: rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
> proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
>        local_lock=none,addr=10.80.111.94
> [root@nfsvmd07 ~]#
>
>
> [root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
> Lck/file: 1, Maxlocks: 10000000
> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
> LOCKED...
>
> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>
> [NFS client successfully locks the file]
>
> On SMB client
> -------------
>
> [root@nfsvme24 ~]# mount |grep cifs
> //nfsvmf24/smb_share on /tmp/mnt type cifs 
> (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
> [root@nfsvme24 ~]#
>
> [root@nfsvme24 ~]# smbclient -L nfsvmf24
> Enter SAMBA\root's password:
>
>     Sharename       Type      Comment
>     ---------       ----      -------
>     print$          Disk      Printer Drivers
>     smb_share       Disk      Test Samba Share       <<===== share to 
> mount
>     IPC$            IPC       IPC Service (Samba 4.10.16)
>     root            Disk      Home Directories
> Reconnecting with SMB1 for workgroup listing.
>
>     Server               Comment
>     ---------            -------
>
>     Workgroup            Master
>     ---------            -------
> [root@nfsvme24 ~]#
>
> [root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
> Lck/file: 1, Maxlocks: 10000000
> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
> LOCKED...
>
> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>
> [SMB client successfully locks the file]
>
> The same issue happens when either client locks the file first.
> I think this is what has happened:
>
> 1. NFSv4 client opens and locks the file first
>
>     . NFSv4 client send OPEN and LOCK to server, server replies
>       OK on both requests.
>
>     . SMB client sends create request with Oplock==Lease for
>       the same file.
>
>     . server holds off on replying to SMB client's create request,
>       recalls delegation from NFSv4 client, waits for NFSv4 client
>       to return the delegation then replies success to SMB client's
>       create request with lease granted (Oplock==Lease).
>
>       NOTE: I think SMB server should replies the create request
>       with Oplock==None to force the SMB client to sends the
>       lock request.
>
>     . Once SMB client receives the reply of the create with
>       'Oplock==Lease', it assumes it has full control of the file
>       therefor it does not need to send the lock request.
>
>     . both NFSv4 and SMB client now think they have locked the file.
>
> pcap:  nfs_lock_smb_lock.pcap
>
> 2. SMB client creates the file with 'Oplock==Lease' first
>
>     . SMB sends create request with 'Oplock==Lease' to server,
>       server replies OK with 'Oplock==Lease'. SMB client skips
>       sending lock request since it assumes it has full control
>       of the file with the lease.
>
>     . NFSv4 client sends OPEN to server, server replies OK with
>       delagation is none. NFSv4 client sends LOCK request, since
>       no lock was created in the kernel for the SMB client, the
>       lock was granted to the NFSv4 client.
>
>      NOTE: I think the SMB server should send lease break
>      notification to the SMB client, wait for the lease break
>      acknowledgment from SMB client before replying to the
>      OPEN of the NFSv4 client. This will force the SMB client
>      to send the lock request to the server.
>
>     . both NFSv4 and SMB client now think they have locked the file.
>
> Your thought?
>
> Thanks,
>
> -Dai
>

[-- Attachment #2: nfs_lock_smb_lock.pcap --]
[-- Type: application/octet-stream, Size: 11324 bytes --]

[-- Attachment #3: smb_lock_nfs_lock.pcap --]
[-- Type: application/octet-stream, Size: 5860 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-07-15 23:45 Locking issue between NFSv4 and SMB client dai.ngo
  2021-07-15 23:47 ` dai.ngo
@ 2021-09-23 21:50 ` Bruce Fields
  2021-09-23 22:39   ` dai.ngo
  2021-09-24  3:35   ` Trond Myklebust
  1 sibling, 2 replies; 15+ messages in thread
From: Bruce Fields @ 2021-09-23 21:50 UTC (permalink / raw)
  To: dai.ngo; +Cc: linux-nfs, CIFS

On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
> Hi Bruce,

Oops, sorry for neglecting this.

> I'm doing some locking testing between NFSv4 and SMB client and
> think there are some issues on the server that allows both clients
> to lock the same file at the same time.

It's not too surprising to me that getting consistent locks between the
two would be hard.

Did you get any review from a Samba expert?  I seem to recall it having
a lot of options, and I wonder if it's configured correctly for this
case.

It sounds like Samba may be giving out oplocks without getting a lease
from the kernel.

--b.

> Here is what I did:
> 
> NOTE: lck is a simple program that use lockf(3) to lock a file from
> offset 0 to the length specified by '-l'.
> 
> On NFSv4 client
> ---------------
> 
> [root@nfsvmd07 ~]# nfsstat -m
> /tmp/mnt from nfsvmf24:/root/smb_share
> Flags:	rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
>        proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
>        local_lock=none,addr=10.80.111.94
> [root@nfsvmd07 ~]#
> 
> 
> [root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
> Lck/file: 1, Maxlocks: 10000000
> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
> LOCKED...
> 
> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
> 
> [NFS client successfully locks the file]
> 
> On SMB client
> -------------
> 
> [root@nfsvme24 ~]# mount |grep cifs
> //nfsvmf24/smb_share on /tmp/mnt type cifs (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
> [root@nfsvme24 ~]#
> 
> [root@nfsvme24 ~]# smbclient -L nfsvmf24
> Enter SAMBA\root's password:
> 
> 	Sharename       Type      Comment
> 	---------       ----      -------
> 	print$          Disk      Printer Drivers
> 	smb_share       Disk      Test Samba Share       <<===== share to mount
> 	IPC$            IPC       IPC Service (Samba 4.10.16)
> 	root            Disk      Home Directories
> Reconnecting with SMB1 for workgroup listing.
> 
> 	Server               Comment
> 	---------            -------
> 
> 	Workgroup            Master
> 	---------            -------
> [root@nfsvme24 ~]#
> 
> [root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
> Lck/file: 1, Maxlocks: 10000000
> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
> LOCKED...
> 
> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
> 
> [SMB client successfully locks the file]
> 
> The same issue happens when either client locks the file first.
> I think this is what has happened:
> 
> 1. NFSv4 client opens and locks the file first
> 
>     . NFSv4 client send OPEN and LOCK to server, server replies
>       OK on both requests.
> 
>     . SMB client sends create request with Oplock==Lease for
>       the same file.
> 
>     . server holds off on replying to SMB client's create request,
>       recalls delegation from NFSv4 client, waits for NFSv4 client
>       to return the delegation then replies success to SMB client's
>       create request with lease granted (Oplock==Lease).
> 
>       NOTE: I think SMB server should replies the create request
>       with Oplock==None to force the SMB client to sends the
>       lock request.
>
>     . Once SMB client receives the reply of the create with
>       'Oplock==Lease', it assumes it has full control of the file
>       therefor it does not need to send the lock request.
> 
>     . both NFSv4 and SMB client now think they have locked the file.
> 
> pcap:  nfs_lock_smb_lock.pcap
> 
> 2. SMB client creates the file with 'Oplock==Lease' first
> 
>     . SMB sends create request with 'Oplock==Lease' to server,
>       server replies OK with 'Oplock==Lease'. SMB client skips
>       sending lock request since it assumes it has full control
>       of the file with the lease.
> 
>     . NFSv4 client sends OPEN to server, server replies OK with
>       delagation is none. NFSv4 client sends LOCK request, since
>       no lock was created in the kernel for the SMB client, the
>       lock was granted to the NFSv4 client.
> 
>      NOTE: I think the SMB server should send lease break
>      notification to the SMB client, wait for the lease break
>      acknowledgment from SMB client before replying to the
>      OPEN of the NFSv4 client. This will force the SMB client
>      to send the lock request to the server.
> 
>     . both NFSv4 and SMB client now think they have locked the file.
> 
> Your thought?
> 
> Thanks,
> 
> -Dai

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-23 21:50 ` Bruce Fields
@ 2021-09-23 22:39   ` dai.ngo
  2021-09-24  0:51     ` Jeremy Allison
  2021-10-07 17:03     ` dai.ngo
  2021-09-24  3:35   ` Trond Myklebust
  1 sibling, 2 replies; 15+ messages in thread
From: dai.ngo @ 2021-09-23 22:39 UTC (permalink / raw)
  To: Bruce Fields; +Cc: linux-nfs, CIFS


On 9/23/21 2:50 PM, Bruce Fields wrote:
> On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
>> Hi Bruce,
> Oops, sorry for neglecting this.
>
>> I'm doing some locking testing between NFSv4 and SMB client and
>> think there are some issues on the server that allows both clients
>> to lock the same file at the same time.
> It's not too surprising to me that getting consistent locks between the
> two would be hard.
>
> Did you get any review from a Samba expert?  I seem to recall it having
> a lot of options, and I wonder if it's configured correctly for this
> case.

No, I have not heard from any Samba expert.

>
> It sounds like Samba may be giving out oplocks without getting a lease
> from the kernel.

I will have to circle back to this when we're done with the 1st
phase of courteous server.

-Dai

>
> --b.
>
>> Here is what I did:
>>
>> NOTE: lck is a simple program that use lockf(3) to lock a file from
>> offset 0 to the length specified by '-l'.
>>
>> On NFSv4 client
>> ---------------
>>
>> [root@nfsvmd07 ~]# nfsstat -m
>> /tmp/mnt from nfsvmf24:/root/smb_share
>> Flags:	rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
>>         proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
>>         local_lock=none,addr=10.80.111.94
>> [root@nfsvmd07 ~]#
>>
>>
>> [root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>> Lck/file: 1, Maxlocks: 10000000
>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
>> LOCKED...
>>
>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>
>> [NFS client successfully locks the file]
>>
>> On SMB client
>> -------------
>>
>> [root@nfsvme24 ~]# mount |grep cifs
>> //nfsvmf24/smb_share on /tmp/mnt type cifs (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
>> [root@nfsvme24 ~]#
>>
>> [root@nfsvme24 ~]# smbclient -L nfsvmf24
>> Enter SAMBA\root's password:
>>
>> 	Sharename       Type      Comment
>> 	---------       ----      -------
>> 	print$          Disk      Printer Drivers
>> 	smb_share       Disk      Test Samba Share       <<===== share to mount
>> 	IPC$            IPC       IPC Service (Samba 4.10.16)
>> 	root            Disk      Home Directories
>> Reconnecting with SMB1 for workgroup listing.
>>
>> 	Server               Comment
>> 	---------            -------
>>
>> 	Workgroup            Master
>> 	---------            -------
>> [root@nfsvme24 ~]#
>>
>> [root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>> Lck/file: 1, Maxlocks: 10000000
>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing F_LOCK..
>> LOCKED...
>>
>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>
>> [SMB client successfully locks the file]
>>
>> The same issue happens when either client locks the file first.
>> I think this is what has happened:
>>
>> 1. NFSv4 client opens and locks the file first
>>
>>      . NFSv4 client send OPEN and LOCK to server, server replies
>>        OK on both requests.
>>
>>      . SMB client sends create request with Oplock==Lease for
>>        the same file.
>>
>>      . server holds off on replying to SMB client's create request,
>>        recalls delegation from NFSv4 client, waits for NFSv4 client
>>        to return the delegation then replies success to SMB client's
>>        create request with lease granted (Oplock==Lease).
>>
>>        NOTE: I think SMB server should replies the create request
>>        with Oplock==None to force the SMB client to sends the
>>        lock request.
>>
>>      . Once SMB client receives the reply of the create with
>>        'Oplock==Lease', it assumes it has full control of the file
>>        therefor it does not need to send the lock request.
>>
>>      . both NFSv4 and SMB client now think they have locked the file.
>>
>> pcap:  nfs_lock_smb_lock.pcap
>>
>> 2. SMB client creates the file with 'Oplock==Lease' first
>>
>>      . SMB sends create request with 'Oplock==Lease' to server,
>>        server replies OK with 'Oplock==Lease'. SMB client skips
>>        sending lock request since it assumes it has full control
>>        of the file with the lease.
>>
>>      . NFSv4 client sends OPEN to server, server replies OK with
>>        delagation is none. NFSv4 client sends LOCK request, since
>>        no lock was created in the kernel for the SMB client, the
>>        lock was granted to the NFSv4 client.
>>
>>       NOTE: I think the SMB server should send lease break
>>       notification to the SMB client, wait for the lease break
>>       acknowledgment from SMB client before replying to the
>>       OPEN of the NFSv4 client. This will force the SMB client
>>       to send the lock request to the server.
>>
>>      . both NFSv4 and SMB client now think they have locked the file.
>>
>> Your thought?
>>
>> Thanks,
>>
>> -Dai

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-23 22:39   ` dai.ngo
@ 2021-09-24  0:51     ` Jeremy Allison
  2021-09-24  0:56       ` Bruce Fields
  2021-10-07 17:03     ` dai.ngo
  1 sibling, 1 reply; 15+ messages in thread
From: Jeremy Allison @ 2021-09-24  0:51 UTC (permalink / raw)
  To: dai.ngo; +Cc: Bruce Fields, linux-nfs, CIFS

On Thu, Sep 23, 2021 at 03:39:52PM -0700, dai.ngo@oracle.com wrote:
>
>On 9/23/21 2:50 PM, Bruce Fields wrote:
>>On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
>>>Hi Bruce,
>>Oops, sorry for neglecting this.
>>
>>>I'm doing some locking testing between NFSv4 and SMB client and
>>>think there are some issues on the server that allows both clients
>>>to lock the same file at the same time.
>>It's not too surprising to me that getting consistent locks between the
>>two would be hard.
>>
>>Did you get any review from a Samba expert?  I seem to recall it having
>>a lot of options, and I wonder if it's configured correctly for this
>>case.
>
>No, I have not heard from any Samba expert.
>
>>
>>It sounds like Samba may be giving out oplocks without getting a lease
>>from the kernel.
>
>I will have to circle back to this when we're done with the 1st
>phase of courteous server.
>
>-Dai
>
>>
>>--b.
>>
>>>Here is what I did:
>>>
>>>NOTE: lck is a simple program that use lockf(3) to lock a file from
>>>offset 0 to the length specified by '-l'.

What does lockf map to in NFS ?

Samba only uses posix fcntl byte range locks (and only when
told to map SMB locks onto underlying posix locks), we don't use
lockf at all.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-24  0:51     ` Jeremy Allison
@ 2021-09-24  0:56       ` Bruce Fields
  0 siblings, 0 replies; 15+ messages in thread
From: Bruce Fields @ 2021-09-24  0:56 UTC (permalink / raw)
  To: Jeremy Allison; +Cc: dai.ngo, linux-nfs, CIFS

On Thu, Sep 23, 2021 at 05:51:13PM -0700, Jeremy Allison wrote:
> On Thu, Sep 23, 2021 at 03:39:52PM -0700, dai.ngo@oracle.com wrote:
> >
> >On 9/23/21 2:50 PM, Bruce Fields wrote:
> >>On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
> >>>Hi Bruce,
> >>Oops, sorry for neglecting this.
> >>
> >>>I'm doing some locking testing between NFSv4 and SMB client and
> >>>think there are some issues on the server that allows both clients
> >>>to lock the same file at the same time.
> >>It's not too surprising to me that getting consistent locks between the
> >>two would be hard.
> >>
> >>Did you get any review from a Samba expert?  I seem to recall it having
> >>a lot of options, and I wonder if it's configured correctly for this
> >>case.
> >
> >No, I have not heard from any Samba expert.
> >
> >>
> >>It sounds like Samba may be giving out oplocks without getting a lease
> >>from the kernel.
> >
> >I will have to circle back to this when we're done with the 1st
> >phase of courteous server.
> >
> >-Dai
> >
> >>
> >>--b.
> >>
> >>>Here is what I did:
> >>>
> >>>NOTE: lck is a simple program that use lockf(3) to lock a file from
> >>>offset 0 to the length specified by '-l'.
> 
> What does lockf map to in NFS ?
> 
> Samba only uses posix fcntl byte range locks (and only when
> told to map SMB locks onto underlying posix locks), we don't use
> lockf at all.

Yeah, it's the same thing, lockf just maps to fcntl locks.

You're probably thinking of flock.

--b.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-23 21:50 ` Bruce Fields
  2021-09-23 22:39   ` dai.ngo
@ 2021-09-24  3:35   ` Trond Myklebust
  2021-09-24  3:46     ` Ralph Boehme
  1 sibling, 1 reply; 15+ messages in thread
From: Trond Myklebust @ 2021-09-24  3:35 UTC (permalink / raw)
  To: bfields, jra, dai.ngo; +Cc: linux-nfs, linux-cifs

On Thu, 2021-09-23 at 17:50 -0400, Bruce Fields wrote:
> On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
> > Hi Bruce,
> 
> Oops, sorry for neglecting this.
> 
> > I'm doing some locking testing between NFSv4 and SMB client and
> > think there are some issues on the server that allows both clients
> > to lock the same file at the same time.
> 
> It's not too surprising to me that getting consistent locks between
> the
> two would be hard.
> 
> Did you get any review from a Samba expert?  I seem to recall it
> having
> a lot of options, and I wonder if it's configured correctly for this
> case.
> 
> It sounds like Samba may be giving out oplocks without getting a
> lease
> from the kernel.
> 

Not if you set the "kernel oplocks" parameter in the smb.conf file. We
just added support for this in the Linux 5.14 kernel NFSv4 client.

Now that said, "kernel oplocks" will currently only support basic level
I oplocks, and cannot support level II or leases. According to the
smb.conf manpage, this is due to some incompleteness in the current VFS
lease implementation.

I'd love to get some more info from the Samba team about what is
missing from the kernel lease implementation that prevents us from
implementing these more advanced oplock/lease features. From the
description in Microsoft's docs, I'm pretty sure that NFSv4 delegations
should be able to provide all the guarantees that are required.

Jeremy, would you be able to elaborate on this topic?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-24  3:35   ` Trond Myklebust
@ 2021-09-24  3:46     ` Ralph Boehme
  2021-09-24  4:13       ` Trond Myklebust
  0 siblings, 1 reply; 15+ messages in thread
From: Ralph Boehme @ 2021-09-24  3:46 UTC (permalink / raw)
  To: Trond Myklebust, bfields, jra, dai.ngo; +Cc: linux-nfs, linux-cifs


[-- Attachment #1.1: Type: text/plain, Size: 1210 bytes --]

Am 24.09.21 um 05:35 schrieb Trond Myklebust:
> Not if you set the "kernel oplocks" parameter in the smb.conf file. We
> just added support for this in the Linux 5.14 kernel NFSv4 client.
> 
> Now that said, "kernel oplocks" will currently only support basic level
> I oplocks, and cannot support level II or leases. According to the
> smb.conf manpage, this is due to some incompleteness in the current VFS
> lease implementation.
> 
> I'd love to get some more info from the Samba team about what is
> missing from the kernel lease implementation that prevents us from
> implementing these more advanced oplock/lease features. From the
> description in Microsoft's docs, I'm pretty sure that NFSv4 delegations
> should be able to provide all the guarantees that are required.

leases can be shared among file handles. When someone requests a lease 
he passes a cookie. Then when he opens the same file with the same 
cookie the lease is not broken.

Maybe others can comment on the level II oplock problem. Afaik this was 
more a lack of testing.

-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-24  3:46     ` Ralph Boehme
@ 2021-09-24  4:13       ` Trond Myklebust
  2021-09-24  4:55         ` Ralph Boehme
  2021-09-24 16:36         ` Jeremy Allison
  0 siblings, 2 replies; 15+ messages in thread
From: Trond Myklebust @ 2021-09-24  4:13 UTC (permalink / raw)
  To: bfields, jra, slow, dai.ngo; +Cc: linux-nfs, linux-cifs

On Fri, 2021-09-24 at 05:46 +0200, Ralph Boehme wrote:
> Am 24.09.21 um 05:35 schrieb Trond Myklebust:
> > Not if you set the "kernel oplocks" parameter in the smb.conf file.
> > We
> > just added support for this in the Linux 5.14 kernel NFSv4 client.
> > 
> > Now that said, "kernel oplocks" will currently only support basic
> > level
> > I oplocks, and cannot support level II or leases. According to the
> > smb.conf manpage, this is due to some incompleteness in the current
> > VFS
> > lease implementation.
> > 
> > I'd love to get some more info from the Samba team about what is
> > missing from the kernel lease implementation that prevents us from
> > implementing these more advanced oplock/lease features. From the
> > description in Microsoft's docs, I'm pretty sure that NFSv4
> > delegations
> > should be able to provide all the guarantees that are required.
> 
> leases can be shared among file handles. When someone requests a
> lease 
> he passes a cookie. Then when he opens the same file with the same 
> cookie the lease is not broken.

Right, but that is easily solved in user space by having the cookie act
as a key that references the file descriptor that holds the lease. This
is how we typically implement NFSv4 delegations as well.

> 
> Maybe others can comment on the level II oplock problem. Afaik this
> was 
> more a lack of testing.

That would really be a bit of a shame, since the use of leases on top
of networked or clustered filesystems could help speed up state
management operations by avoiding the need for a separate distributed
lock manager.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-24  4:13       ` Trond Myklebust
@ 2021-09-24  4:55         ` Ralph Boehme
  2021-09-24 16:36         ` Jeremy Allison
  1 sibling, 0 replies; 15+ messages in thread
From: Ralph Boehme @ 2021-09-24  4:55 UTC (permalink / raw)
  To: Trond Myklebust, bfields, jra, dai.ngo; +Cc: linux-nfs, linux-cifs


[-- Attachment #1.1: Type: text/plain, Size: 768 bytes --]

Am 24.09.21 um 06:13 schrieb Trond Myklebust:
> On Fri, 2021-09-24 at 05:46 +0200, Ralph Boehme wrote:
>> leases can be shared among file handles. When someone requests a
>> lease
>> he passes a cookie. Then when he opens the same file with the same
>> cookie the lease is not broken.
> 
> Right, but that is easily solved in user space by having the cookie act
> as a key that references the file descriptor that holds the lease. This
> is how we typically implement NFSv4 delegations as well
yeah. We already track the required state in userspace anyway, so this 
should be a low hanging fruit (more or less :) ).

-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-24  4:13       ` Trond Myklebust
  2021-09-24  4:55         ` Ralph Boehme
@ 2021-09-24 16:36         ` Jeremy Allison
  2021-09-24 16:45           ` bfields
  1 sibling, 1 reply; 15+ messages in thread
From: Jeremy Allison @ 2021-09-24 16:36 UTC (permalink / raw)
  To: Trond Myklebust; +Cc: bfields, slow, dai.ngo, linux-nfs, linux-cifs

On Fri, Sep 24, 2021 at 04:13:23AM +0000, Trond Myklebust wrote:
>On Fri, 2021-09-24 at 05:46 +0200, Ralph Boehme wrote:
>> Am 24.09.21 um 05:35 schrieb Trond Myklebust:
>> > Not if you set the "kernel oplocks" parameter in the smb.conf file.
>> > We
>> > just added support for this in the Linux 5.14 kernel NFSv4 client.
>> >
>> > Now that said, "kernel oplocks" will currently only support basic
>> > level
>> > I oplocks, and cannot support level II or leases. According to the
>> > smb.conf manpage, this is due to some incompleteness in the current
>> > VFS
>> > lease implementation.
>> >
>> > I'd love to get some more info from the Samba team about what is
>> > missing from the kernel lease implementation that prevents us from
>> > implementing these more advanced oplock/lease features. From the
>> > description in Microsoft's docs, I'm pretty sure that NFSv4
>> > delegations
>> > should be able to provide all the guarantees that are required.
>>
>> leases can be shared among file handles. When someone requests a
>> lease
>> he passes a cookie. Then when he opens the same file with the same
>> cookie the lease is not broken.
>
>Right, but that is easily solved in user space by having the cookie act
>as a key that references the file descriptor that holds the lease. This
>is how we typically implement NFSv4 delegations as well.

How does this work in multi-process situations ?
When you say "file descriptor", if the fd was passed
between processes would the lease state transfer ?

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-24 16:36         ` Jeremy Allison
@ 2021-09-24 16:45           ` bfields
  0 siblings, 0 replies; 15+ messages in thread
From: bfields @ 2021-09-24 16:45 UTC (permalink / raw)
  To: Jeremy Allison; +Cc: Trond Myklebust, slow, dai.ngo, linux-nfs, linux-cifs

On Fri, Sep 24, 2021 at 09:36:46AM -0700, Jeremy Allison wrote:
> On Fri, Sep 24, 2021 at 04:13:23AM +0000, Trond Myklebust wrote:
> >On Fri, 2021-09-24 at 05:46 +0200, Ralph Boehme wrote:
> >>Am 24.09.21 um 05:35 schrieb Trond Myklebust:
> >>> Not if you set the "kernel oplocks" parameter in the smb.conf file.
> >>> We
> >>> just added support for this in the Linux 5.14 kernel NFSv4 client.
> >>>
> >>> Now that said, "kernel oplocks" will currently only support basic
> >>> level
> >>> I oplocks, and cannot support level II or leases. According to the
> >>> smb.conf manpage, this is due to some incompleteness in the current
> >>> VFS
> >>> lease implementation.
> >>>
> >>> I'd love to get some more info from the Samba team about what is
> >>> missing from the kernel lease implementation that prevents us from
> >>> implementing these more advanced oplock/lease features. From the
> >>> description in Microsoft's docs, I'm pretty sure that NFSv4
> >>> delegations
> >>> should be able to provide all the guarantees that are required.
> >>
> >>leases can be shared among file handles. When someone requests a
> >>lease
> >>he passes a cookie. Then when he opens the same file with the same
> >>cookie the lease is not broken.
> >
> >Right, but that is easily solved in user space by having the cookie act
> >as a key that references the file descriptor that holds the lease. This
> >is how we typically implement NFSv4 delegations as well.
> 
> How does this work in multi-process situations ?
> When you say "file descriptor", if the fd was passed
> between processes would the lease state transfer ?

Yes, the lease is associated with the "file description"/struct file, so
for example you should be able to F_UNLCK it from the new process.

--b.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-09-23 22:39   ` dai.ngo
  2021-09-24  0:51     ` Jeremy Allison
@ 2021-10-07 17:03     ` dai.ngo
  2021-10-07 17:38       ` dai.ngo
  1 sibling, 1 reply; 15+ messages in thread
From: dai.ngo @ 2021-10-07 17:03 UTC (permalink / raw)
  To: Bruce Fields; +Cc: linux-nfs, CIFS


On 9/23/21 3:39 PM, dai.ngo@oracle.com wrote:
>
> On 9/23/21 2:50 PM, Bruce Fields wrote:
>> On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
>>> Hi Bruce,
>> Oops, sorry for neglecting this.
>>
>>> I'm doing some locking testing between NFSv4 and SMB client and
>>> think there are some issues on the server that allows both clients
>>> to lock the same file at the same time.
>> It's not too surprising to me that getting consistent locks between the
>> two would be hard.
>>
>> Did you get any review from a Samba expert?  I seem to recall it having
>> a lot of options, and I wonder if it's configured correctly for this
>> case.
>
> No, I have not heard from any Samba expert.
>
>>
>> It sounds like Samba may be giving out oplocks without getting a lease
>> from the kernel.
>
> I will have to circle back to this when we're done with the 1st
> phase of courteous server.

I disabled oplock for the SMB share and locking between NFSv4 and SMB
client works as expected. It appears that smbd does not set the VFS
lease on the file after granting oplock to smb client.

-Dai

> -Dai
>
>>
>> --b.
>>
>>> Here is what I did:
>>>
>>> NOTE: lck is a simple program that use lockf(3) to lock a file from
>>> offset 0 to the length specified by '-l'.
>>>
>>> On NFSv4 client
>>> ---------------
>>>
>>> [root@nfsvmd07 ~]# nfsstat -m
>>> /tmp/mnt from nfsvmf24:/root/smb_share
>>> Flags: 
>>> rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
>>> proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
>>>         local_lock=none,addr=10.80.111.94
>>> [root@nfsvmd07 ~]#
>>>
>>>
>>> [root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>> Lck/file: 1, Maxlocks: 10000000
>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>> F_LOCK..
>>> LOCKED...
>>>
>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>
>>> [NFS client successfully locks the file]
>>>
>>> On SMB client
>>> -------------
>>>
>>> [root@nfsvme24 ~]# mount |grep cifs
>>> //nfsvmf24/smb_share on /tmp/mnt type cifs 
>>> (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
>>> [root@nfsvme24 ~]#
>>>
>>> [root@nfsvme24 ~]# smbclient -L nfsvmf24
>>> Enter SAMBA\root's password:
>>>
>>>     Sharename       Type      Comment
>>>     ---------       ----      -------
>>>     print$          Disk      Printer Drivers
>>>     smb_share       Disk      Test Samba Share <<===== share to mount
>>>     IPC$            IPC       IPC Service (Samba 4.10.16)
>>>     root            Disk      Home Directories
>>> Reconnecting with SMB1 for workgroup listing.
>>>
>>>     Server               Comment
>>>     ---------            -------
>>>
>>>     Workgroup            Master
>>>     ---------            -------
>>> [root@nfsvme24 ~]#
>>>
>>> [root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>> Lck/file: 1, Maxlocks: 10000000
>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>> F_LOCK..
>>> LOCKED...
>>>
>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>
>>> [SMB client successfully locks the file]
>>>
>>> The same issue happens when either client locks the file first.
>>> I think this is what has happened:
>>>
>>> 1. NFSv4 client opens and locks the file first
>>>
>>>      . NFSv4 client send OPEN and LOCK to server, server replies
>>>        OK on both requests.
>>>
>>>      . SMB client sends create request with Oplock==Lease for
>>>        the same file.
>>>
>>>      . server holds off on replying to SMB client's create request,
>>>        recalls delegation from NFSv4 client, waits for NFSv4 client
>>>        to return the delegation then replies success to SMB client's
>>>        create request with lease granted (Oplock==Lease).
>>>
>>>        NOTE: I think SMB server should replies the create request
>>>        with Oplock==None to force the SMB client to sends the
>>>        lock request.
>>>
>>>      . Once SMB client receives the reply of the create with
>>>        'Oplock==Lease', it assumes it has full control of the file
>>>        therefor it does not need to send the lock request.
>>>
>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>
>>> pcap:  nfs_lock_smb_lock.pcap
>>>
>>> 2. SMB client creates the file with 'Oplock==Lease' first
>>>
>>>      . SMB sends create request with 'Oplock==Lease' to server,
>>>        server replies OK with 'Oplock==Lease'. SMB client skips
>>>        sending lock request since it assumes it has full control
>>>        of the file with the lease.
>>>
>>>      . NFSv4 client sends OPEN to server, server replies OK with
>>>        delagation is none. NFSv4 client sends LOCK request, since
>>>        no lock was created in the kernel for the SMB client, the
>>>        lock was granted to the NFSv4 client.
>>>
>>>       NOTE: I think the SMB server should send lease break
>>>       notification to the SMB client, wait for the lease break
>>>       acknowledgment from SMB client before replying to the
>>>       OPEN of the NFSv4 client. This will force the SMB client
>>>       to send the lock request to the server.
>>>
>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>
>>> Your thought?
>>>
>>> Thanks,
>>>
>>> -Dai

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-10-07 17:03     ` dai.ngo
@ 2021-10-07 17:38       ` dai.ngo
  2021-10-11 16:21         ` Bruce Fields
  0 siblings, 1 reply; 15+ messages in thread
From: dai.ngo @ 2021-10-07 17:38 UTC (permalink / raw)
  To: Bruce Fields; +Cc: linux-nfs, CIFS


On 10/7/21 10:03 AM, dai.ngo@oracle.com wrote:
>
> On 9/23/21 3:39 PM, dai.ngo@oracle.com wrote:
>>
>> On 9/23/21 2:50 PM, Bruce Fields wrote:
>>> On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
>>>> Hi Bruce,
>>> Oops, sorry for neglecting this.
>>>
>>>> I'm doing some locking testing between NFSv4 and SMB client and
>>>> think there are some issues on the server that allows both clients
>>>> to lock the same file at the same time.
>>> It's not too surprising to me that getting consistent locks between the
>>> two would be hard.
>>>
>>> Did you get any review from a Samba expert?  I seem to recall it having
>>> a lot of options, and I wonder if it's configured correctly for this
>>> case.
>>
>> No, I have not heard from any Samba expert.
>>
>>>
>>> It sounds like Samba may be giving out oplocks without getting a lease
>>> from the kernel.
>>
>> I will have to circle back to this when we're done with the 1st
>> phase of courteous server.
>
> I disabled oplock for the SMB share and locking between NFSv4 and SMB
> client works as expected. It appears that smbd does not set the VFS
> lease on the file after granting oplock to smb client.

Enabling kernel oplocks has the same effect, smbd does not grant oplock
to client forcing it to send lock request.

-Dai

>
> -Dai
>
>> -Dai
>>
>>>
>>> --b.
>>>
>>>> Here is what I did:
>>>>
>>>> NOTE: lck is a simple program that use lockf(3) to lock a file from
>>>> offset 0 to the length specified by '-l'.
>>>>
>>>> On NFSv4 client
>>>> ---------------
>>>>
>>>> [root@nfsvmd07 ~]# nfsstat -m
>>>> /tmp/mnt from nfsvmf24:/root/smb_share
>>>> Flags: 
>>>> rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
>>>> proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
>>>>         local_lock=none,addr=10.80.111.94
>>>> [root@nfsvmd07 ~]#
>>>>
>>>>
>>>> [root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>>> Lck/file: 1, Maxlocks: 10000000
>>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>>> F_LOCK..
>>>> LOCKED...
>>>>
>>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>>
>>>> [NFS client successfully locks the file]
>>>>
>>>> On SMB client
>>>> -------------
>>>>
>>>> [root@nfsvme24 ~]# mount |grep cifs
>>>> //nfsvmf24/smb_share on /tmp/mnt type cifs 
>>>> (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
>>>> [root@nfsvme24 ~]#
>>>>
>>>> [root@nfsvme24 ~]# smbclient -L nfsvmf24
>>>> Enter SAMBA\root's password:
>>>>
>>>>     Sharename       Type      Comment
>>>>     ---------       ----      -------
>>>>     print$          Disk      Printer Drivers
>>>>     smb_share       Disk      Test Samba Share <<===== share to mount
>>>>     IPC$            IPC       IPC Service (Samba 4.10.16)
>>>>     root            Disk      Home Directories
>>>> Reconnecting with SMB1 for workgroup listing.
>>>>
>>>>     Server               Comment
>>>>     ---------            -------
>>>>
>>>>     Workgroup            Master
>>>>     ---------            -------
>>>> [root@nfsvme24 ~]#
>>>>
>>>> [root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>>> Lck/file: 1, Maxlocks: 10000000
>>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>>> F_LOCK..
>>>> LOCKED...
>>>>
>>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>>
>>>> [SMB client successfully locks the file]
>>>>
>>>> The same issue happens when either client locks the file first.
>>>> I think this is what has happened:
>>>>
>>>> 1. NFSv4 client opens and locks the file first
>>>>
>>>>      . NFSv4 client send OPEN and LOCK to server, server replies
>>>>        OK on both requests.
>>>>
>>>>      . SMB client sends create request with Oplock==Lease for
>>>>        the same file.
>>>>
>>>>      . server holds off on replying to SMB client's create request,
>>>>        recalls delegation from NFSv4 client, waits for NFSv4 client
>>>>        to return the delegation then replies success to SMB client's
>>>>        create request with lease granted (Oplock==Lease).
>>>>
>>>>        NOTE: I think SMB server should replies the create request
>>>>        with Oplock==None to force the SMB client to sends the
>>>>        lock request.
>>>>
>>>>      . Once SMB client receives the reply of the create with
>>>>        'Oplock==Lease', it assumes it has full control of the file
>>>>        therefor it does not need to send the lock request.
>>>>
>>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>>
>>>> pcap:  nfs_lock_smb_lock.pcap
>>>>
>>>> 2. SMB client creates the file with 'Oplock==Lease' first
>>>>
>>>>      . SMB sends create request with 'Oplock==Lease' to server,
>>>>        server replies OK with 'Oplock==Lease'. SMB client skips
>>>>        sending lock request since it assumes it has full control
>>>>        of the file with the lease.
>>>>
>>>>      . NFSv4 client sends OPEN to server, server replies OK with
>>>>        delagation is none. NFSv4 client sends LOCK request, since
>>>>        no lock was created in the kernel for the SMB client, the
>>>>        lock was granted to the NFSv4 client.
>>>>
>>>>       NOTE: I think the SMB server should send lease break
>>>>       notification to the SMB client, wait for the lease break
>>>>       acknowledgment from SMB client before replying to the
>>>>       OPEN of the NFSv4 client. This will force the SMB client
>>>>       to send the lock request to the server.
>>>>
>>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>>
>>>> Your thought?
>>>>
>>>> Thanks,
>>>>
>>>> -Dai

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Locking issue between NFSv4 and SMB client
  2021-10-07 17:38       ` dai.ngo
@ 2021-10-11 16:21         ` Bruce Fields
  0 siblings, 0 replies; 15+ messages in thread
From: Bruce Fields @ 2021-10-11 16:21 UTC (permalink / raw)
  To: dai.ngo; +Cc: linux-nfs, CIFS

On Thu, Oct 07, 2021 at 10:38:45AM -0700, dai.ngo@oracle.com wrote:
> 
> On 10/7/21 10:03 AM, dai.ngo@oracle.com wrote:
> >
> >On 9/23/21 3:39 PM, dai.ngo@oracle.com wrote:
> >>
> >>On 9/23/21 2:50 PM, Bruce Fields wrote:
> >>>On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
> >>>>Hi Bruce,
> >>>Oops, sorry for neglecting this.
> >>>
> >>>>I'm doing some locking testing between NFSv4 and SMB client and
> >>>>think there are some issues on the server that allows both clients
> >>>>to lock the same file at the same time.
> >>>It's not too surprising to me that getting consistent locks between the
> >>>two would be hard.
> >>>
> >>>Did you get any review from a Samba expert?  I seem to recall it having
> >>>a lot of options, and I wonder if it's configured correctly for this
> >>>case.
> >>
> >>No, I have not heard from any Samba expert.
> >>
> >>>
> >>>It sounds like Samba may be giving out oplocks without getting a lease
> >>>from the kernel.
> >>
> >>I will have to circle back to this when we're done with the 1st
> >>phase of courteous server.
> >
> >I disabled oplock for the SMB share and locking between NFSv4 and SMB
> >client works as expected. It appears that smbd does not set the VFS
> >lease on the file after granting oplock to smb client.
> 
> Enabling kernel oplocks has the same effect, smbd does not grant oplock
> to client forcing it to send lock request.

OK, good, so that's working as expected.

I understand that there are still some deficiencies in the kernel lease
implementation, but I'm not sure how to hit those cases with this kind
of testing.

--b.

> 
> -Dai
> 
> >
> >-Dai
> >
> >>-Dai
> >>
> >>>
> >>>--b.
> >>>
> >>>>Here is what I did:
> >>>>
> >>>>NOTE: lck is a simple program that use lockf(3) to lock a file from
> >>>>offset 0 to the length specified by '-l'.
> >>>>
> >>>>On NFSv4 client
> >>>>---------------
> >>>>
> >>>>[root@nfsvmd07 ~]# nfsstat -m
> >>>>/tmp/mnt from nfsvmf24:/root/smb_share
> >>>>Flags:
> >>>>rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
> >>>>proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
> >>>>        local_lock=none,addr=10.80.111.94
> >>>>[root@nfsvmd07 ~]#
> >>>>
> >>>>
> >>>>[root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
> >>>>Lck/file: 1, Maxlocks: 10000000
> >>>>Locking[/tmp/mnt/messages] Offset[0] Len[100000000]
> >>>>N[0]...doing F_LOCK..
> >>>>LOCKED...
> >>>>
> >>>>Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
> >>>>
> >>>>[NFS client successfully locks the file]
> >>>>
> >>>>On SMB client
> >>>>-------------
> >>>>
> >>>>[root@nfsvme24 ~]# mount |grep cifs
> >>>>//nfsvmf24/smb_share on /tmp/mnt type cifs (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
> >>>>[root@nfsvme24 ~]#
> >>>>
> >>>>[root@nfsvme24 ~]# smbclient -L nfsvmf24
> >>>>Enter SAMBA\root's password:
> >>>>
> >>>>    Sharename       Type      Comment
> >>>>    ---------       ----      -------
> >>>>    print$          Disk      Printer Drivers
> >>>>    smb_share       Disk      Test Samba Share <<===== share to mount
> >>>>    IPC$            IPC       IPC Service (Samba 4.10.16)
> >>>>    root            Disk      Home Directories
> >>>>Reconnecting with SMB1 for workgroup listing.
> >>>>
> >>>>    Server               Comment
> >>>>    ---------            -------
> >>>>
> >>>>    Workgroup            Master
> >>>>    ---------            -------
> >>>>[root@nfsvme24 ~]#
> >>>>
> >>>>[root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
> >>>>Lck/file: 1, Maxlocks: 10000000
> >>>>Locking[/tmp/mnt/messages] Offset[0] Len[100000000]
> >>>>N[0]...doing F_LOCK..
> >>>>LOCKED...
> >>>>
> >>>>Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
> >>>>
> >>>>[SMB client successfully locks the file]
> >>>>
> >>>>The same issue happens when either client locks the file first.
> >>>>I think this is what has happened:
> >>>>
> >>>>1. NFSv4 client opens and locks the file first
> >>>>
> >>>>     . NFSv4 client send OPEN and LOCK to server, server replies
> >>>>       OK on both requests.
> >>>>
> >>>>     . SMB client sends create request with Oplock==Lease for
> >>>>       the same file.
> >>>>
> >>>>     . server holds off on replying to SMB client's create request,
> >>>>       recalls delegation from NFSv4 client, waits for NFSv4 client
> >>>>       to return the delegation then replies success to SMB client's
> >>>>       create request with lease granted (Oplock==Lease).
> >>>>
> >>>>       NOTE: I think SMB server should replies the create request
> >>>>       with Oplock==None to force the SMB client to sends the
> >>>>       lock request.
> >>>>
> >>>>     . Once SMB client receives the reply of the create with
> >>>>       'Oplock==Lease', it assumes it has full control of the file
> >>>>       therefor it does not need to send the lock request.
> >>>>
> >>>>     . both NFSv4 and SMB client now think they have locked the file.
> >>>>
> >>>>pcap:  nfs_lock_smb_lock.pcap
> >>>>
> >>>>2. SMB client creates the file with 'Oplock==Lease' first
> >>>>
> >>>>     . SMB sends create request with 'Oplock==Lease' to server,
> >>>>       server replies OK with 'Oplock==Lease'. SMB client skips
> >>>>       sending lock request since it assumes it has full control
> >>>>       of the file with the lease.
> >>>>
> >>>>     . NFSv4 client sends OPEN to server, server replies OK with
> >>>>       delagation is none. NFSv4 client sends LOCK request, since
> >>>>       no lock was created in the kernel for the SMB client, the
> >>>>       lock was granted to the NFSv4 client.
> >>>>
> >>>>      NOTE: I think the SMB server should send lease break
> >>>>      notification to the SMB client, wait for the lease break
> >>>>      acknowledgment from SMB client before replying to the
> >>>>      OPEN of the NFSv4 client. This will force the SMB client
> >>>>      to send the lock request to the server.
> >>>>
> >>>>     . both NFSv4 and SMB client now think they have locked the file.
> >>>>
> >>>>Your thought?
> >>>>
> >>>>Thanks,
> >>>>
> >>>>-Dai

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2021-10-11 16:21 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-15 23:45 Locking issue between NFSv4 and SMB client dai.ngo
2021-07-15 23:47 ` dai.ngo
2021-09-23 21:50 ` Bruce Fields
2021-09-23 22:39   ` dai.ngo
2021-09-24  0:51     ` Jeremy Allison
2021-09-24  0:56       ` Bruce Fields
2021-10-07 17:03     ` dai.ngo
2021-10-07 17:38       ` dai.ngo
2021-10-11 16:21         ` Bruce Fields
2021-09-24  3:35   ` Trond Myklebust
2021-09-24  3:46     ` Ralph Boehme
2021-09-24  4:13       ` Trond Myklebust
2021-09-24  4:55         ` Ralph Boehme
2021-09-24 16:36         ` Jeremy Allison
2021-09-24 16:45           ` bfields

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).