* pNFS: invalid IP:port selection when talks to DS
@ 2017-03-20 15:52 Mkrtchyan, Tigran
2017-03-20 16:16 ` Mkrtchyan, Tigran
2017-03-20 20:14 ` Olga Kornievskaia
0 siblings, 2 replies; 7+ messages in thread
From: Mkrtchyan, Tigran @ 2017-03-20 15:52 UTC (permalink / raw)
To: Linux NFS Mailing list; +Cc: Steve Dickson
[-- Attachment #1: Type: text/plain, Size: 3244 bytes --]
Dear (p)NFS-ors,
we observe VERY unpleasant situation with pNFS in the production.
Our hosts run multiple DSes on different ports, usually 24001-24009.
With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes
a wrong port number when talks to data server:
If client uses different DSes on the same host, then at some point it starts
to send data to the wrong port number:
Client <=> MDS:
1 0.000000000 131.169.251.53 → 131.169.51.35 NFS V4 Call OPEN DH: 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
2 0.001469799 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 1) OPEN StateID: 0xec18
3 0.001578128 131.169.251.53 → 131.169.51.35 NFS V4 Call SETATTR FH: 0x6ccf3dfa
4 0.002657187 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 3) SETATTR
5 0.003243819 131.169.251.53 → 131.169.51.35 NFS V4 Call LAYOUTGET
6 0.014603386 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 5) LAYOUTGET
7 0.014899121 131.169.251.53 → 131.169.51.35 NFS V4 Call GETDEVINFO
8 0.015014216 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 7) GETDEVINFO
Opcode: GETDEVINFO (47)
Status: NFS4_OK (0)
layout type: LAYOUT4_NFSV4_1_FILES (1)
device index: 0
r_netid: tcp
length: 3
contents: tcp
fill bytes: opaque data
r_addr: 131.169.51.50.93.197
length: 20
contents: 131.169.51.50.93.197
r_netid: tcp
length: 3
contents: tcp
fill bytes: opaque data
r_addr: 131.169.51.50.93.197
length: 20
contents: 131.169.51.50.93.197
notification bitmap: 6
notification bitmap: 0
[Main Opcode: GETDEVINFO (47)]
9 0.105442455 131.169.251.53 → 131.169.51.35 NFS V4 Call TEST_STATEID
10 0.105521354 131.169.51.35 → 131.169.251.53 NFS V4 Reply (Call In 9) TEST_STATEID
NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
client <=> DS
$ tshark -r ds-write.pcap -n -z conv,tcp
1 0.000000 131.169.251.53 → 131.169.51.50 NFS V4 Call WRITE StateID: 0xff01 Offset: 0 Len: 3968
2 0.000090 131.169.51.50 → 131.169.251.53 NFS V4 Reply (Call In 1) WRITE Status: NFS4ERR_BAD_STATEID
================================================================================
TCP Conversations
Filter:<No Filter>
| <- | | -> | | Total | Relative | Duration |
| Frames Bytes | | Frames Bytes | | Frames Bytes | Start | |
131.169.51.50:24006 <-> 131.169.251.53:847 1 4240 1 168 2 4408 0.000000000 0.0001
================================================================================
NOTICE, that it talks to DS on port 24006!
Is there know fix which is missing in CentOS7? I can't reproduce it with
4.9 kernel (or it's harder to reproduce).
The packages are attached.
Tigran.
[-- Attachment #2: ds-write.pcapng --]
[-- Type: application/x-pcapng, Size: 4520 bytes --]
[-- Attachment #3: mds.pcapng --]
[-- Type: application/x-pcapng, Size: 3580 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pNFS: invalid IP:port selection when talks to DS
2017-03-20 15:52 pNFS: invalid IP:port selection when talks to DS Mkrtchyan, Tigran
@ 2017-03-20 16:16 ` Mkrtchyan, Tigran
2017-03-20 20:14 ` Olga Kornievskaia
1 sibling, 0 replies; 7+ messages in thread
From: Mkrtchyan, Tigran @ 2017-03-20 16:16 UTC (permalink / raw)
To: Linux NFS Mailing list; +Cc: Steve Dickson
re-sending without attachments.
The capture failes can be found at:
client <-> mds: https://desycloud.desy.de/index.php/s/58JFyfMQmNF99pU
client <-> ds: https://desycloud.desy.de/index.php/s/dKf290ikQcifL9K
Tigran.
----- Original Message -----
> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
> To: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>
> Cc: "Steve Dickson" <steved@redhat.com>
> Sent: Monday, March 20, 2017 4:52:40 PM
> Subject: pNFS: invalid IP:port selection when talks to DS
> Dear (p)NFS-ors,
>=20
> we observe VERY unpleasant situation with pNFS in the production.
> Our hosts run multiple DSes on different ports, usually 24001-24009.
> With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes
> a wrong port number when talks to data server:
>=20
> If client uses different DSes on the same host, then at some point it sta=
rts
> to send data to the wrong port number:
>=20
> Client <=3D> MDS:
>=20
>=20
> 1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OPEN =
DH:
> 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
> 2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=
l In 1) OPEN
> StateID: 0xec18
> 3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SETAT=
TR FH: 0x6ccf3dfa
> 4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=
l In 3) SETATTR
> 5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LAYOU=
TGET
> 6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=
l In 5) LAYOUTGET
> 7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GETDE=
VINFO
> 8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=
l In 7) GETDEVINFO
> Opcode: GETDEVINFO (47)
> Status: NFS4_OK (0)
> layout type: LAYOUT4_NFSV4_1_FILES (1)
> device index: 0
> r_netid: tcp
> length: 3
> contents: tcp
> fill bytes: opaque data
> r_addr: 131.169.51.50.93.197
> length: 20
> contents: 131.169.51.50.93.197
> r_netid: tcp
> length: 3
> contents: tcp
> fill bytes: opaque data
> r_addr: 131.169.51.50.93.197
> length: 20
> contents: 131.169.51.50.93.197
> notification bitmap: 6
> notification bitmap: 0
> [Main Opcode: GETDEVINFO (47)]
>=20
> 9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TEST_=
STATEID
> 10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=
l In 9)
> TEST_STATEID
>=20
>=20
>=20
> NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
>=20
> client <=3D> DS
>=20
> $ tshark -r ds-write.pcap -n -z conv,tcp
> 1 0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRITE =
StateID: 0xff01
> Offset: 0 Len: 3968
> 2 0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (Call=
In 1) WRITE
> Status: NFS4ERR_BAD_STATEID
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> TCP Conversations
> Filter:<No Filter>
> | <- =
| | -> | | Total | Relative | Duration |
> | Frames Bytes=
| | Frames Bytes | | Frames Bytes | Start |
> | |
> 131.169.51.50:24006 <-> 131.169.251.53:847 1 42=
40
> 1 168 2 4408 0.000000000 0.0001
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
>=20
> NOTICE, that it talks to DS on port 24006!
>=20
> Is there know fix which is missing in CentOS7? I can't reproduce it with
> 4.9 kernel (or it's harder to reproduce).
>=20
>=20
> The packages are attached.
>=20
> Tigran.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pNFS: invalid IP:port selection when talks to DS
2017-03-20 15:52 pNFS: invalid IP:port selection when talks to DS Mkrtchyan, Tigran
2017-03-20 16:16 ` Mkrtchyan, Tigran
@ 2017-03-20 20:14 ` Olga Kornievskaia
2017-03-20 20:51 ` Mkrtchyan, Tigran
1 sibling, 1 reply; 7+ messages in thread
From: Olga Kornievskaia @ 2017-03-20 20:14 UTC (permalink / raw)
To: Mkrtchyan, Tigran; +Cc: Linux NFS Mailing list, Steve Dickson
Hi Tigran,
While I don't have an answer to your question, I'd like to point out
that in 4.9 is when Andy's session trunking patches when in.
I'm curious this client that's now talking to the DS at port 24006
instead of 24005, did it before also earlier correctly (legally)
talked to DS that was on 24006?
On Mon, Mar 20, 2017 at 11:52 AM, Mkrtchyan, Tigran
<tigran.mkrtchyan@desy.de> wrote:
>
>
> Dear (p)NFS-ors,
>
> we observe VERY unpleasant situation with pNFS in the production.
> Our hosts run multiple DSes on different ports, usually 24001-24009.
> With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes
> a wrong port number when talks to data server:
>
> If client uses different DSes on the same host, then at some point it sta=
rts
> to send data to the wrong port number:
>
> Client <=3D> MDS:
>
>
> 1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OPEN=
DH: 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
> 2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=
ll In 1) OPEN StateID: 0xec18
> 3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SETA=
TTR FH: 0x6ccf3dfa
> 4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=
ll In 3) SETATTR
> 5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LAYO=
UTGET
> 6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=
ll In 5) LAYOUTGET
> 7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GETD=
EVINFO
> 8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=
ll In 7) GETDEVINFO
> Opcode: GETDEVINFO (47)
> Status: NFS4_OK (0)
> layout type: LAYOUT4_NFSV4_1_FILES (1)
> device index: 0
> r_netid: tcp
> length: 3
> contents: tcp
> fill bytes: opaque data
> r_addr: 131.169.51.50.93.197
> length: 20
> contents: 131.169.51.50.93.197
> r_netid: tcp
> length: 3
> contents: tcp
> fill bytes: opaque data
> r_addr: 131.169.51.50.93.197
> length: 20
> contents: 131.169.51.50.93.197
> notification bitmap: 6
> notification bitmap: 0
> [Main Opcode: GETDEVINFO (47)]
>
> 9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TEST=
_STATEID
> 10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=
ll In 9) TEST_STATEID
>
>
>
> NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
>
> client <=3D> DS
>
> $ tshark -r ds-write.pcap -n -z conv,tcp
> 1 0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRITE=
StateID: 0xff01 Offset: 0 Len: 3968
> 2 0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (Cal=
l In 1) WRITE Status: NFS4ERR_BAD_STATEID
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> TCP Conversations
> Filter:<No Filter>
> | <- =
| | -> | | Total | Relative | Duration |
> | Frames Byte=
s | | Frames Bytes | | Frames Bytes | Start | |
> 131.169.51.50:24006 <-> 131.169.251.53:847 1 42=
40 1 168 2 4408 0.000000000 0.0001
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
>
> NOTICE, that it talks to DS on port 24006!
>
> Is there know fix which is missing in CentOS7? I can't reproduce it with
> 4.9 kernel (or it's harder to reproduce).
>
>
> The packages are attached.
>
> Tigran.
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pNFS: invalid IP:port selection when talks to DS
2017-03-20 20:14 ` Olga Kornievskaia
@ 2017-03-20 20:51 ` Mkrtchyan, Tigran
2017-03-20 21:09 ` Mkrtchyan, Tigran
0 siblings, 1 reply; 7+ messages in thread
From: Mkrtchyan, Tigran @ 2017-03-20 20:51 UTC (permalink / raw)
To: Olga Kornievskaia; +Cc: Linux NFS Mailing list, Steve Dickson
Hi Olga,
----- Original Message -----
> From: "Olga Kornievskaia" <aglo@umich.edu>
> To: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dickson"=
<steved@redhat.com>
> Sent: Monday, March 20, 2017 9:14:34 PM
> Subject: Re: pNFS: invalid IP:port selection when talks to DS
> Hi Tigran,
>=20
> While I don't have an answer to your question, I'd like to point out
> that in 4.9 is when Andy's session trunking patches when in.
>=20
> I'm curious this client that's now talking to the DS at port 24006
> instead of 24005, did it before also earlier correctly (legally)
> talked to DS that was on 24006?
Yes, earlier during testing it had legal access to DS on port 24006.
Tigran.
>=20
> On Mon, Mar 20, 2017 at 11:52 AM, Mkrtchyan, Tigran
> <tigran.mkrtchyan@desy.de> wrote:
>>
>>
>> Dear (p)NFS-ors,
>>
>> we observe VERY unpleasant situation with pNFS in the production.
>> Our hosts run multiple DSes on different ports, usually 24001-24009.
>> With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes
>> a wrong port number when talks to data server:
>>
>> If client uses different DSes on the same host, then at some point it st=
arts
>> to send data to the wrong port number:
>>
>> Client <=3D> MDS:
>>
>>
>> 1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OPE=
N DH:
>> 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
>> 2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=
all In 1) OPEN
>> StateID: 0xec18
>> 3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SET=
ATTR FH: 0x6ccf3dfa
>> 4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=
all In 3) SETATTR
>> 5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LAY=
OUTGET
>> 6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=
all In 5) LAYOUTGET
>> 7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GET=
DEVINFO
>> 8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=
all In 7) GETDEVINFO
>> Opcode: GETDEVINFO (47)
>> Status: NFS4_OK (0)
>> layout type: LAYOUT4_NFSV4_1_FILES (1)
>> device index: 0
>> r_netid: tcp
>> length: 3
>> contents: tcp
>> fill bytes: opaque data
>> r_addr: 131.169.51.50.93.197
>> length: 20
>> contents: 131.169.51.50.93.197
>> r_netid: tcp
>> length: 3
>> contents: tcp
>> fill bytes: opaque data
>> r_addr: 131.169.51.50.93.197
>> length: 20
>> contents: 131.169.51.50.93.197
>> notification bitmap: 6
>> notification bitmap: 0
>> [Main Opcode: GETDEVINFO (47)]
>>
>> 9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TES=
T_STATEID
>> 10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (C=
all In 9)
>> TEST_STATEID
>>
>>
>>
>> NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
>>
>> client <=3D> DS
>>
>> $ tshark -r ds-write.pcap -n -z conv,tcp
>> 1 0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRIT=
E StateID: 0xff01
>> Offset: 0 Len: 3968
>> 2 0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (Ca=
ll In 1) WRITE
>> Status: NFS4ERR_BAD_STATEID
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
>> TCP Conversations
>> Filter:<No Filter>
>> | <- =
| | -> | | Total | Relative | Duration |
>> | Frames Byt=
es | | Frames Bytes | | Frames Bytes | Start |
>> | |
>> 131.169.51.50:24006 <-> 131.169.251.53:847 1 4=
240
>> 1 168 2 4408 0.000000000 0.0001
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
>>
>> NOTICE, that it talks to DS on port 24006!
>>
>> Is there know fix which is missing in CentOS7? I can't reproduce it with
>> 4.9 kernel (or it's harder to reproduce).
>>
>>
>> The packages are attached.
>>
>> Tigran.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pNFS: invalid IP:port selection when talks to DS
2017-03-20 20:51 ` Mkrtchyan, Tigran
@ 2017-03-20 21:09 ` Mkrtchyan, Tigran
2017-03-22 16:04 ` Olga Kornievskaia
0 siblings, 1 reply; 7+ messages in thread
From: Mkrtchyan, Tigran @ 2017-03-20 21:09 UTC (permalink / raw)
To: Olga Kornievskaia; +Cc: Linux NFS Mailing list, Steve Dickson
Hi Olga,
you did not have the answer, however you gave me an important hint!
I believe, all our DSes on a single host generate the same server
owner during exchange-id. I guess, this can be the reason, why
client decides to talk to an other DS.
Tigran.
----- Original Message -----
> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
> To: "Olga Kornievskaia" <aglo@umich.edu>
> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dickson"=
<steved@redhat.com>
> Sent: Monday, March 20, 2017 9:51:21 PM
> Subject: Re: pNFS: invalid IP:port selection when talks to DS
> Hi Olga,
>=20
> ----- Original Message -----
>> From: "Olga Kornievskaia" <aglo@umich.edu>
>> To: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
>> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dickson=
"
>> <steved@redhat.com>
>> Sent: Monday, March 20, 2017 9:14:34 PM
>> Subject: Re: pNFS: invalid IP:port selection when talks to DS
>=20
>> Hi Tigran,
>>=20
>> While I don't have an answer to your question, I'd like to point out
>> that in 4.9 is when Andy's session trunking patches when in.
>>=20
>> I'm curious this client that's now talking to the DS at port 24006
>> instead of 24005, did it before also earlier correctly (legally)
>> talked to DS that was on 24006?
>=20
> Yes, earlier during testing it had legal access to DS on port 24006.
>=20
> Tigran.
>=20
>>=20
>> On Mon, Mar 20, 2017 at 11:52 AM, Mkrtchyan, Tigran
>> <tigran.mkrtchyan@desy.de> wrote:
>>>
>>>
>>> Dear (p)NFS-ors,
>>>
>>> we observe VERY unpleasant situation with pNFS in the production.
>>> Our hosts run multiple DSes on different ports, usually 24001-24009.
>>> With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes
>>> a wrong port number when talks to data server:
>>>
>>> If client uses different DSes on the same host, then at some point it s=
tarts
>>> to send data to the wrong port number:
>>>
>>> Client <=3D> MDS:
>>>
>>>
>>> 1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call OP=
EN DH:
>>> 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
>>> 2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=
Call In 1) OPEN
>>> StateID: 0xec18
>>> 3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call SE=
TATTR FH: 0x6ccf3dfa
>>> 4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=
Call In 3) SETATTR
>>> 5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call LA=
YOUTGET
>>> 6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=
Call In 5) LAYOUTGET
>>> 7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call GE=
TDEVINFO
>>> 8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=
Call In 7) GETDEVINFO
>>> Opcode: GETDEVINFO (47)
>>> Status: NFS4_OK (0)
>>> layout type: LAYOUT4_NFSV4_1_FILES (1)
>>> device index: 0
>>> r_netid: tcp
>>> length: 3
>>> contents: tcp
>>> fill bytes: opaque data
>>> r_addr: 131.169.51.50.93.197
>>> length: 20
>>> contents: 131.169.51.50.93.197
>>> r_netid: tcp
>>> length: 3
>>> contents: tcp
>>> fill bytes: opaque data
>>> r_addr: 131.169.51.50.93.197
>>> length: 20
>>> contents: 131.169.51.50.93.197
>>> notification bitmap: 6
>>> notification bitmap: 0
>>> [Main Opcode: GETDEVINFO (47)]
>>>
>>> 9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call TE=
ST_STATEID
>>> 10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply (=
Call In 9)
>>> TEST_STATEID
>>>
>>>
>>>
>>> NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
>>>
>>> client <=3D> DS
>>>
>>> $ tshark -r ds-write.pcap -n -z conv,tcp
>>> 1 0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WRI=
TE StateID: 0xff01
>>> Offset: 0 Len: 3968
>>> 2 0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (C=
all In 1) WRITE
>>> Status: NFS4ERR_BAD_STATEID
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>>> TCP Conversations
>>> Filter:<No Filter>
>>> | <- =
| | -> | | Total | Relative | Duration |
>>> | Frames By=
tes | | Frames Bytes | | Frames Bytes | Start |
>>> | |
>>> 131.169.51.50:24006 <-> 131.169.251.53:847 1 =
4240
>>> 1 168 2 4408 0.000000000 0.0001
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>>>
>>> NOTICE, that it talks to DS on port 24006!
>>>
>>> Is there know fix which is missing in CentOS7? I can't reproduce it wit=
h
>>> 4.9 kernel (or it's harder to reproduce).
>>>
>>>
>>> The packages are attached.
>>>
>>> Tigran.
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pNFS: invalid IP:port selection when talks to DS
2017-03-20 21:09 ` Mkrtchyan, Tigran
@ 2017-03-22 16:04 ` Olga Kornievskaia
2017-03-22 20:27 ` Mkrtchyan, Tigran
0 siblings, 1 reply; 7+ messages in thread
From: Olga Kornievskaia @ 2017-03-22 16:04 UTC (permalink / raw)
To: Mkrtchyan, Tigran; +Cc: Linux NFS Mailing list, Steve Dickson
Hi Tigran,
I still don't have the answer to your question but I'm just puzzled
why it "works" with 4.9 (session trunking). New code would check the
server owner and if they are the same, then it would add that to the
list of addresses to trunk. I'd assume you'd be seeing the same
behavior with the new code. Thus, I'm puzzled. That aside, if you
don't want the new code to trunk between your DSs on the same server,
they should return different owner.
I'm assuming device ids are different for the DSs on different ports?
On Mon, Mar 20, 2017 at 5:09 PM, Mkrtchyan, Tigran
<tigran.mkrtchyan@desy.de> wrote:
>
> Hi Olga,
>
> you did not have the answer, however you gave me an important hint!
> I believe, all our DSes on a single host generate the same server
> owner during exchange-id. I guess, this can be the reason, why
> client decides to talk to an other DS.
>
> Tigran.
>
> ----- Original Message -----
>> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
>> To: "Olga Kornievskaia" <aglo@umich.edu>
>> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dickson=
" <steved@redhat.com>
>> Sent: Monday, March 20, 2017 9:51:21 PM
>> Subject: Re: pNFS: invalid IP:port selection when talks to DS
>
>> Hi Olga,
>>
>> ----- Original Message -----
>>> From: "Olga Kornievskaia" <aglo@umich.edu>
>>> To: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
>>> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dickso=
n"
>>> <steved@redhat.com>
>>> Sent: Monday, March 20, 2017 9:14:34 PM
>>> Subject: Re: pNFS: invalid IP:port selection when talks to DS
>>
>>> Hi Tigran,
>>>
>>> While I don't have an answer to your question, I'd like to point out
>>> that in 4.9 is when Andy's session trunking patches when in.
>>>
>>> I'm curious this client that's now talking to the DS at port 24006
>>> instead of 24005, did it before also earlier correctly (legally)
>>> talked to DS that was on 24006?
>>
>> Yes, earlier during testing it had legal access to DS on port 24006.
>>
>> Tigran.
>>
>>>
>>> On Mon, Mar 20, 2017 at 11:52 AM, Mkrtchyan, Tigran
>>> <tigran.mkrtchyan@desy.de> wrote:
>>>>
>>>>
>>>> Dear (p)NFS-ors,
>>>>
>>>> we observe VERY unpleasant situation with pNFS in the production.
>>>> Our hosts run multiple DSes on different ports, usually 24001-24009.
>>>> With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes
>>>> a wrong port number when talks to data server:
>>>>
>>>> If client uses different DSes on the same host, then at some point it =
starts
>>>> to send data to the wrong port number:
>>>>
>>>> Client <=3D> MDS:
>>>>
>>>>
>>>> 1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call O=
PEN DH:
>>>> 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
>>>> 2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =
(Call In 1) OPEN
>>>> StateID: 0xec18
>>>> 3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call S=
ETATTR FH: 0x6ccf3dfa
>>>> 4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =
(Call In 3) SETATTR
>>>> 5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call L=
AYOUTGET
>>>> 6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =
(Call In 5) LAYOUTGET
>>>> 7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call G=
ETDEVINFO
>>>> 8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =
(Call In 7) GETDEVINFO
>>>> Opcode: GETDEVINFO (47)
>>>> Status: NFS4_OK (0)
>>>> layout type: LAYOUT4_NFSV4_1_FILES (1)
>>>> device index: 0
>>>> r_netid: tcp
>>>> length: 3
>>>> contents: tcp
>>>> fill bytes: opaque data
>>>> r_addr: 131.169.51.50.93.197
>>>> length: 20
>>>> contents: 131.169.51.50.93.197
>>>> r_netid: tcp
>>>> length: 3
>>>> contents: tcp
>>>> fill bytes: opaque data
>>>> r_addr: 131.169.51.50.93.197
>>>> length: 20
>>>> contents: 131.169.51.50.93.197
>>>> notification bitmap: 6
>>>> notification bitmap: 0
>>>> [Main Opcode: GETDEVINFO (47)]
>>>>
>>>> 9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call T=
EST_STATEID
>>>> 10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply =
(Call In 9)
>>>> TEST_STATEID
>>>>
>>>>
>>>>
>>>> NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
>>>>
>>>> client <=3D> DS
>>>>
>>>> $ tshark -r ds-write.pcap -n -z conv,tcp
>>>> 1 0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call WR=
ITE StateID: 0xff01
>>>> Offset: 0 Len: 3968
>>>> 2 0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply (=
Call In 1) WRITE
>>>> Status: NFS4ERR_BAD_STATEID
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>>>> TCP Conversations
>>>> Filter:<No Filter>
>>>> | <- =
| | -> | | Total | Relative | Duration |
>>>> | Frames B=
ytes | | Frames Bytes | | Frames Bytes | Start |
>>>> | |
>>>> 131.169.51.50:24006 <-> 131.169.251.53:847 1 =
4240
>>>> 1 168 2 4408 0.000000000 0.0001
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>>>>
>>>> NOTICE, that it talks to DS on port 24006!
>>>>
>>>> Is there know fix which is missing in CentOS7? I can't reproduce it wi=
th
>>>> 4.9 kernel (or it's harder to reproduce).
>>>>
>>>>
>>>> The packages are attached.
>>>>
>>>> Tigran.
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pNFS: invalid IP:port selection when talks to DS
2017-03-22 16:04 ` Olga Kornievskaia
@ 2017-03-22 20:27 ` Mkrtchyan, Tigran
0 siblings, 0 replies; 7+ messages in thread
From: Mkrtchyan, Tigran @ 2017-03-22 20:27 UTC (permalink / raw)
To: Olga Kornievskaia; +Cc: Linux NFS Mailing list
Hi Olga,
----- Original Message -----
> From: "Olga Kornievskaia" <aglo@umich.edu>
> To: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dickson"=
<steved@redhat.com>
> Sent: Wednesday, March 22, 2017 5:04:21 PM
> Subject: Re: pNFS: invalid IP:port selection when talks to DS
> Hi Tigran,
>=20
> I still don't have the answer to your question but I'm just puzzled
> why it "works" with 4.9 (session trunking). New code would check the
> server owner and if they are the same, then it would add that to the
> list of addresses to trunk. I'd assume you'd be seeing the same
> behavior with the new code. Thus, I'm puzzled. That aside, if you
> don't want the new code to trunk between your DSs on the same server,
> they should return different owner.
I have no idea why 4.9 works differently. May be this no enough 'load'
to trigger trunking. Anyway, I have updated our code to generate different
server owners. We still need to test it, before all 600 DSes get updated.
>=20
> I'm assuming device ids are different for the DSs on different ports?
Yes, each DS gets a unique id.
Thanks,
Tigran.
>=20
> On Mon, Mar 20, 2017 at 5:09 PM, Mkrtchyan, Tigran
> <tigran.mkrtchyan@desy.de> wrote:
>>
>> Hi Olga,
>>
>> you did not have the answer, however you gave me an important hint!
>> I believe, all our DSes on a single host generate the same server
>> owner during exchange-id. I guess, this can be the reason, why
>> client decides to talk to an other DS.
>>
>> Tigran.
>>
>> ----- Original Message -----
>>> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
>>> To: "Olga Kornievskaia" <aglo@umich.edu>
>>> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dickso=
n"
>>> <steved@redhat.com>
>>> Sent: Monday, March 20, 2017 9:51:21 PM
>>> Subject: Re: pNFS: invalid IP:port selection when talks to DS
>>
>>> Hi Olga,
>>>
>>> ----- Original Message -----
>>>> From: "Olga Kornievskaia" <aglo@umich.edu>
>>>> To: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
>>>> Cc: "Linux NFS Mailing list" <linux-nfs@vger.kernel.org>, "Steve Dicks=
on"
>>>> <steved@redhat.com>
>>>> Sent: Monday, March 20, 2017 9:14:34 PM
>>>> Subject: Re: pNFS: invalid IP:port selection when talks to DS
>>>
>>>> Hi Tigran,
>>>>
>>>> While I don't have an answer to your question, I'd like to point out
>>>> that in 4.9 is when Andy's session trunking patches when in.
>>>>
>>>> I'm curious this client that's now talking to the DS at port 24006
>>>> instead of 24005, did it before also earlier correctly (legally)
>>>> talked to DS that was on 24006?
>>>
>>> Yes, earlier during testing it had legal access to DS on port 24006.
>>>
>>> Tigran.
>>>
>>>>
>>>> On Mon, Mar 20, 2017 at 11:52 AM, Mkrtchyan, Tigran
>>>> <tigran.mkrtchyan@desy.de> wrote:
>>>>>
>>>>>
>>>>> Dear (p)NFS-ors,
>>>>>
>>>>> we observe VERY unpleasant situation with pNFS in the production.
>>>>> Our hosts run multiple DSes on different ports, usually 24001-24009.
>>>>> With CentOS7 (3.10.0-514.6.2.el7.x86_64) we see that client takes
>>>>> a wrong port number when talks to data server:
>>>>>
>>>>> If client uses different DSes on the same host, then at some point it=
starts
>>>>> to send data to the wrong port number:
>>>>>
>>>>> Client <=3D> MDS:
>>>>>
>>>>>
>>>>> 1 0.000000000 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =
OPEN DH:
>>>>> 0x7cbc716b/MIL-68-onebatch-80C-30s-00057.tif.metadata
>>>>> 2 0.001469799 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=
(Call In 1) OPEN
>>>>> StateID: 0xec18
>>>>> 3 0.001578128 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =
SETATTR FH: 0x6ccf3dfa
>>>>> 4 0.002657187 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=
(Call In 3) SETATTR
>>>>> 5 0.003243819 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =
LAYOUTGET
>>>>> 6 0.014603386 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=
(Call In 5) LAYOUTGET
>>>>> 7 0.014899121 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =
GETDEVINFO
>>>>> 8 0.015014216 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=
(Call In 7) GETDEVINFO
>>>>> Opcode: GETDEVINFO (47)
>>>>> Status: NFS4_OK (0)
>>>>> layout type: LAYOUT4_NFSV4_1_FILES (1)
>>>>> device index: 0
>>>>> r_netid: tcp
>>>>> length: 3
>>>>> contents: tcp
>>>>> fill bytes: opaque data
>>>>> r_addr: 131.169.51.50.93.197
>>>>> length: 20
>>>>> contents: 131.169.51.50.93.197
>>>>> r_netid: tcp
>>>>> length: 3
>>>>> contents: tcp
>>>>> fill bytes: opaque data
>>>>> r_addr: 131.169.51.50.93.197
>>>>> length: 20
>>>>> contents: 131.169.51.50.93.197
>>>>> notification bitmap: 6
>>>>> notification bitmap: 0
>>>>> [Main Opcode: GETDEVINFO (47)]
>>>>>
>>>>> 9 0.105442455 131.169.251.53 =E2=86=92 131.169.51.35 NFS V4 Call =
TEST_STATEID
>>>>> 10 0.105521354 131.169.51.35 =E2=86=92 131.169.251.53 NFS V4 Reply=
(Call In 9)
>>>>> TEST_STATEID
>>>>>
>>>>>
>>>>>
>>>>> NOTICE, that 131.169.51.50.93.197 corresponds to port 24005.
>>>>>
>>>>> client <=3D> DS
>>>>>
>>>>> $ tshark -r ds-write.pcap -n -z conv,tcp
>>>>> 1 0.000000 131.169.251.53 =E2=86=92 131.169.51.50 NFS V4 Call W=
RITE StateID: 0xff01
>>>>> Offset: 0 Len: 3968
>>>>> 2 0.000090 131.169.51.50 =E2=86=92 131.169.251.53 NFS V4 Reply =
(Call In 1) WRITE
>>>>> Status: NFS4ERR_BAD_STATEID
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>>>>> TCP Conversations
>>>>> Filter:<No Filter>
>>>>> | <-=
| | -> | | Total | Relative | Duration |
>>>>> | Frames =
Bytes | | Frames Bytes | | Frames Bytes | Start |
>>>>> | |
>>>>> 131.169.51.50:24006 <-> 131.169.251.53:847 1 =
4240
>>>>> 1 168 2 4408 0.000000000 0.0001
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>>>>>
>>>>> NOTICE, that it talks to DS on port 24006!
>>>>>
>>>>> Is there know fix which is missing in CentOS7? I can't reproduce it w=
ith
>>>>> 4.9 kernel (or it's harder to reproduce).
>>>>>
>>>>>
>>>>> The packages are attached.
>>>>>
>>>>> Tigran.
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" i=
n
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-03-22 20:27 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-20 15:52 pNFS: invalid IP:port selection when talks to DS Mkrtchyan, Tigran
2017-03-20 16:16 ` Mkrtchyan, Tigran
2017-03-20 20:14 ` Olga Kornievskaia
2017-03-20 20:51 ` Mkrtchyan, Tigran
2017-03-20 21:09 ` Mkrtchyan, Tigran
2017-03-22 16:04 ` Olga Kornievskaia
2017-03-22 20:27 ` Mkrtchyan, Tigran
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.