From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
To: trondmy <trondmy@hammerspace.com>
Cc: linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS file+flexfiles data channels
Date: Sat, 14 Nov 2020 00:46:52 +0100 (CET) [thread overview]
Message-ID: <1371149886.691555.1605311212511.JavaMail.zimbra@desy.de> (raw)
In-Reply-To: <d73c15ca631ad52f036bb8708ab15b89af432952.camel@hammerspace.com>
----- Original Message -----
> From: "trondmy" <trondmy@hammerspace.com>
> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>
> Cc: "linux-nfs" <linux-nfs@vger.kernel.org>
> Sent: Friday, 13 November, 2020 23:45:00
> Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS file+flexfiles data channels
> On Fri, 2020-11-13 at 22:30 +0100, Mkrtchyan, Tigran wrote:
>>
>> After more testing, it looks like that client doesn't like
>> notification bitmap:
>>
>>
>> [31576.789492] --> _nfs4_proc_getdeviceinfo
>> [31576.789503] --> nfs41_call_sync_prepare data->seq_server
>> 000000001d17c43e
>> [31576.789507] --> nfs4_alloc_slot used_slots=0000
>> highest_used=4294967295 max_slots=16
>> [31576.789510] <-- nfs4_alloc_slot used_slots=0001 highest_used=0
>> slotid=0
>> [31576.789527] encode_sequence:
>> sessionid=2910695007:150995712:0:16777216 seqid=92462 slotid=0
>> max_slotid=0 cache_this=0
>> [31576.789991] decode_getdeviceinfo: unsupported notification
>
> According to this, you appear to be returning a deviceinfo bitmap with
> at least one non-zero entry that is not in the first 32-bit word. We
> only ask for notifications for NOTIFY_DEVICEID4_CHANGE and
> NOTIFY_DEVICEID4_DELETE, so we only expect bitmap[0] to have non-zero
> entries.
according to packet capture only bitmap[0] has non zero bits set.
This is the reply of compound starting from nfs staus code, tag
length and so on.
0000 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 35
0010 00 00 00 00 5f ae 7d ad 00 03 00 09 00 00 00 00
0020 00 00 00 01 00 00 00 4c 00 00 00 00 00 00 00 0f
0030 00 00 00 0f 00 00 00 00 00 00 00 2f 00 00 00 00
0040 00 00 00 04 00 00 00 40 00 00 00 01 00 00 00 03
0050 74 63 70 00 00 00 00 16 31 33 31 2e 31 36 39 2e
0060 31 39 31 2e 31 34 33 2e 31 32 35 2e 34 39 00 00
0070 00 00 00 01 00 00 00 04 00 00 00 01 00 10 00 00
0080 00 10 00 00 00 00 00 01 00 00 00 02 00 00 00 06
0090 00 00 00 00
the last 12 bytes : bitmap size, bitmap[0], bitmap[1]
This part of code in the didn't change since 2010, and I
have no issues to use 5.8 kernel. I am pretty sure, that
tests with 5.9 did pass as expected. I will try to bisec it.
Tigran.
>
>> [31576.790003] --> nfs4_alloc_slot used_slots=0001 highest_used=0
>> max_slots=16
>> [31576.790007] <-- nfs4_alloc_slot used_slots=0003 highest_used=1
>> slotid=1
>> [31576.790010] nfs4_free_slot: slotid 1 highest_used_slotid 0
>> [31576.790013] nfs41_sequence_process: Error 0 free the slot
>> [31576.790017] nfs4_free_slot: slotid 0 highest_used_slotid
>> 4294967295
>> [31576.790030] <-- _nfs4_proc_getdeviceinfo status=-5
>> [31576.790034] nfs4_get_device_info getdevice info returns -5
>> [31576.790084] <-- nfs4_get_device_info d 0000000000000000
>>
>>
>> Tigran.
>>
>>
>> ----- Original Message -----
>> > From: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>
>> > To: "trondmy" <trondmy@hammerspace.com>
>> > Cc: "linux-nfs" <linux-nfs@vger.kernel.org>
>> > Sent: Friday, 13 November, 2020 13:48:07
>> > Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS
>> > file+flexfiles data channels
>>
>> > Hi Trond,
>> >
>> > whit this changes (3ee69a14f92d74ced2647140b3799511ba4f3fa5) I see
>> > an infinite
>> > loop
>> > of LAYOUTGET->GETDEVICEINFO->LAYOUTRETURN without any attempt to
>> > connect to a
>> > DS.
>> >
>> > This is how the response to LAYOUTGET looks like.
>> >
>> > Network File System, Ops(2): SEQUENCE GETDEVINFO
>> > [Program Version: 4]
>> > [V4 Procedure: COMPOUND (1)]
>> > Status: NFS4_OK (0)
>> > Tag: <EMPTY>
>> > length: 0
>> > contents: <EMPTY>
>> > Operations (count: 2)
>> > Opcode: SEQUENCE (53)
>> > Opcode: GETDEVINFO (47)
>> > Status: NFS4_OK (0)
>> > layout type: LAYOUT4_FLEX_FILES (4)
>> > r_netid: tcp
>> > length: 3
>> > contents: tcp
>> > fill bytes: opaque data
>> > r_addr: 131.169.191.143.125.49
>> > length: 22
>> > contents: 131.169.191.143.125.49
>> > fill bytes: opaque data
>> > version: 4
>> > minorversion: 1
>> > max_rsize: 1048576
>> > max_wsize: 1048576
>> > tightly coupled: Yes
>> > notify_mask: 0x00000006 (Change, Delete)
>> > notify_type: Change (1)
>> > notify_type: Delete (2)
>> > [Main Opcode: GETDEVINFO (47)]
>> >
>> >
>> >
>> > The MDS is mounted with IPv4. I can provide the full packet trace,
>> > if needed.
>> >
>> >
>> > Regards,
>> > Tigran.
>> >
>> >
>> > ----- Original Message -----
>> > > From: "trondmy" <trondmy@hammerspace.com>
>> > > To: "linux-nfs" <linux-nfs@vger.kernel.org>
>> > > Sent: Wednesday, 11 November, 2020 00:42:31
>> > > Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS
>> > > file+flexfiles data
>> > > channels
>> >
>> > > On Tue, 2020-11-10 at 18:18 -0500, trondmy@kernel.org wrote:
>> > > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
>> > > >
>> > > > Add support for connecting to the pNFS files/flexfiles data
>> > > > servers
>> > > > through RDMA, assuming that the GETDEVICEINFO call advertises
>> > > > that
>> > > > support.
>> > > >
>> > > > v2: Fix layoutstats encoding for pNFS/flexfiles.
>> > > > v3: Move most of the netid handling into the SUNRPC and RDMA
>> > > > modules.
>> > > > Fix up the mount code to benefit more from automated
>> > > > loading of
>> > > > SUNRPC transport modules.
>> > > >
>> > >
>> > > Note that one cleanup that I did not perform, but which really
>> > > could be
>> > > useful should we want to add more transport mechanisms, is to
>> > > move the
>> > > code to parse stringified addresses (in particular IETF style
>> > > "universal addresses") into the transport modules so that the
>> > > actual
>> > > parsing of mount and pNFS transport info can be automatically
>> > > extended
>> > > when new transport modules are added.
>> > >
>> > > --
>> > > Trond Myklebust
>> > > Linux NFS client maintainer, Hammerspace
>> > > trond.myklebust@hammerspace.com
>
> --
> Trond Myklebust
> CTO, Hammerspace Inc
> 4984 El Camino Real, Suite 208
> Los Altos, CA 94022
>
> www.hammer.space
next prev parent reply other threads:[~2020-11-13 23:47 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-10 23:18 [PATCH v3 00/11] Add RDMA support to the pNFS file+flexfiles data channels trondmy
2020-11-10 23:18 ` [PATCH v3 01/11] SUNRPC: xprt_load_transport() needs to support the netid "rdma6" trondmy
2020-11-10 23:18 ` [PATCH v3 02/11] SUNRPC: Close a race with transport setup and module put trondmy
2020-11-10 23:18 ` [PATCH v3 03/11] SUNRPC: Add a helper to return the transport identifier given a netid trondmy
2020-11-10 23:18 ` [PATCH v3 04/11] NFS: Switch mount code to use xprt_find_transport_ident() trondmy
2020-11-10 23:19 ` [PATCH v3 05/11] SUNRPC: Remove unused function xprt_load_transport() trondmy
2020-11-10 23:19 ` [PATCH v3 06/11] NFSv4/pNFS: Use connections to a DS that are all of the same protocol family trondmy
2020-11-10 23:19 ` [PATCH v3 07/11] pNFS: Add helpers for allocation/free of struct nfs4_pnfs_ds_addr trondmy
2020-11-10 23:19 ` [PATCH v3 08/11] NFSv4/pNFS: Store the transport type in " trondmy
2020-11-10 23:19 ` [PATCH v3 09/11] pNFS/flexfiles: Fix up layoutstats reporting for non-TCP transports trondmy
2020-11-10 23:19 ` [PATCH v3 10/11] SUNRPC: Fix up open coded kmemdup_nul() trondmy
2020-11-10 23:19 ` [PATCH v3 11/11] pNFS: Clean up open coded xdr string decoding trondmy
2020-11-10 23:42 ` [PATCH v3 00/11] Add RDMA support to the pNFS file+flexfiles data channels Trond Myklebust
2020-11-13 12:48 ` Mkrtchyan, Tigran
2020-11-13 21:30 ` Mkrtchyan, Tigran
2020-11-13 22:45 ` Trond Myklebust
2020-11-13 23:46 ` Mkrtchyan, Tigran [this message]
2020-11-14 14:29 ` Trond Myklebust
2020-11-16 20:55 ` Mkrtchyan, Tigran
2020-11-17 14:50 ` Mkrtchyan, Tigran
2020-11-26 17:17 ` Mkrtchyan, Tigran
2020-12-01 10:59 ` Mkrtchyan, Tigran
2020-12-01 14:44 ` Mkrtchyan, Tigran
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1371149886.691555.1605311212511.JavaMail.zimbra@desy.de \
--to=tigran.mkrtchyan@desy.de \
--cc=linux-nfs@vger.kernel.org \
--cc=trondmy@hammerspace.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.