Re: [PATCH v3 00/11] Add RDMA support to the pNFS file+flexfiles data channels

[Trond Myklebust <trondmy@hammerspace.com>]

On Fri, 2020-11-13 at 22:30 +0100, Mkrtchyan, Tigran wrote:
> 
> After more testing, it looks like that client doesn't like
> notification bitmap:
> 
> 
> [31576.789492] --> _nfs4_proc_getdeviceinfo
> [31576.789503] --> nfs41_call_sync_prepare data->seq_server
> 000000001d17c43e
> [31576.789507] --> nfs4_alloc_slot used_slots=0000
> highest_used=4294967295 max_slots=16
> [31576.789510] <-- nfs4_alloc_slot used_slots=0001 highest_used=0
> slotid=0
> [31576.789527] encode_sequence:
> sessionid=2910695007:150995712:0:16777216 seqid=92462 slotid=0
> max_slotid=0 cache_this=0
> [31576.789991] decode_getdeviceinfo: unsupported notification

According to this, you appear to be returning a deviceinfo bitmap with
at least one non-zero entry that is not in the first 32-bit word. We
only ask for notifications for NOTIFY_DEVICEID4_CHANGE and
NOTIFY_DEVICEID4_DELETE, so we only expect bitmap[0] to have non-zero
entries.

> [31576.790003] --> nfs4_alloc_slot used_slots=0001 highest_used=0
> max_slots=16
> [31576.790007] <-- nfs4_alloc_slot used_slots=0003 highest_used=1
> slotid=1
> [31576.790010] nfs4_free_slot: slotid 1 highest_used_slotid 0
> [31576.790013] nfs41_sequence_process: Error 0 free the slot
> [31576.790017] nfs4_free_slot: slotid 0 highest_used_slotid
> 4294967295
> [31576.790030] <-- _nfs4_proc_getdeviceinfo status=-5
> [31576.790034] nfs4_get_device_info getdevice info returns -5
> [31576.790084] <-- nfs4_get_device_info d 0000000000000000
> 
> 
> Tigran.
> 
> 
> ----- Original Message -----
> > From: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>
> > To: "trondmy" <trondmy@hammerspace.com>
> > Cc: "linux-nfs" <linux-nfs@vger.kernel.org>
> > Sent: Friday, 13 November, 2020 13:48:07
> > Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS
> > file+flexfiles data channels
> 
> > Hi Trond,
> > 
> > whit this changes (3ee69a14f92d74ced2647140b3799511ba4f3fa5) I see
> > an infinite
> > loop
> > of LAYOUTGET->GETDEVICEINFO->LAYOUTRETURN without any attempt to
> > connect to a
> > DS.
> > 
> > This is how the response to LAYOUTGET looks like.
> > 
> > Network File System, Ops(2): SEQUENCE GETDEVINFO
> >    [Program Version: 4]
> >    [V4 Procedure: COMPOUND (1)]
> >    Status: NFS4_OK (0)
> >    Tag: <EMPTY>
> >        length: 0
> >        contents: <EMPTY>
> >    Operations (count: 2)
> >        Opcode: SEQUENCE (53)
> >        Opcode: GETDEVINFO (47)
> >            Status: NFS4_OK (0)
> >            layout type: LAYOUT4_FLEX_FILES (4)
> >            r_netid: tcp
> >                length: 3
> >                contents: tcp
> >                fill bytes: opaque data
> >            r_addr: 131.169.191.143.125.49
> >                length: 22
> >                contents: 131.169.191.143.125.49
> >                fill bytes: opaque data
> >            version: 4
> >            minorversion: 1
> >            max_rsize: 1048576
> >            max_wsize: 1048576
> >            tightly coupled: Yes
> >            notify_mask: 0x00000006 (Change, Delete)
> >                notify_type: Change (1)
> >                notify_type: Delete (2)
> >    [Main Opcode: GETDEVINFO (47)]
> > 
> > 
> > 
> > The MDS is mounted with IPv4. I can provide the full packet trace,
> > if needed.
> > 
> > 
> > Regards,
> >   Tigran.
> > 
> > 
> > ----- Original Message -----
> > > From: "trondmy" <trondmy@hammerspace.com>
> > > To: "linux-nfs" <linux-nfs@vger.kernel.org>
> > > Sent: Wednesday, 11 November, 2020 00:42:31
> > > Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS
> > > file+flexfiles data
> > > channels
> > 
> > > On Tue, 2020-11-10 at 18:18 -0500, trondmy@kernel.org wrote:
> > > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > 
> > > > Add support for connecting to the pNFS files/flexfiles data
> > > > servers
> > > > through RDMA, assuming that the GETDEVICEINFO call advertises
> > > > that
> > > > support.
> > > > 
> > > > v2: Fix layoutstats encoding for pNFS/flexfiles.
> > > > v3: Move most of the netid handling into the SUNRPC and RDMA
> > > > modules.
> > > >     Fix up the mount code to benefit more from automated
> > > > loading of
> > > >     SUNRPC transport modules.
> > > > 
> > > 
> > > Note that one cleanup that I did not perform, but which really
> > > could be
> > > useful should we want to add more transport mechanisms, is to
> > > move the
> > > code to parse stringified addresses (in particular IETF style
> > > "universal addresses") into the transport modules so that the
> > > actual
> > > parsing of mount and pNFS transport info can be automatically
> > > extended
> > > when new transport modules are added.
> > > 
> > > --
> > > Trond Myklebust
> > > Linux NFS client maintainer, Hammerspace
> > > trond.myklebust@hammerspace.com

-- 
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022
​
www.hammer.space