pNFS, multiple paths to Data Server not used by pNFS linux client.

pNFS, multiple paths to Data Server not used by pNFS linux client.

[Aditya Agnihotri]

Hi

We are testing our pNFS server with linux client, when the server returns m=
ultiple paths to a v4 DS via GETDEVINFO, we see that the client validated t=
he paths (does exchange id) but does not use the alternate paths in IO.
Tracing the code path, it seems that when DS is a NFS v3 server, path is ad=
ded, but for NFS v4 DS server, client setup function callback path fails to=
 add.

Forcing the paths to be added seems to fix the issue, so we think there mig=
ht be a bug.

We see that when data server is v3, it's setup function does call
rpc_xprt_switch_add_xprt

=20
via , static const struct rpc_call_ops rpc_cb_add_xprt_call_ops =3D {
        .rpc_call_done =3D rpc_cb_add_xprt_done,
        .rpc_release =3D rpc_cb_add_xprt_release,
};

=20
nfs4_pnfs_ds_connect->_nfs4_pnfs_v3_ds_connect->
rpc_clnt_add_xprt->
rpc_clnt_test_and_add_xprt->
rpc_cb_add_xprt_done->rpc_xprt_switch_add_xprt

but v4 add xprt does not, so a connection for multipathing gets validated f=
or v4, but never added.

nfs4_pnfs_ds_connect->nfs4_pnfs_ds_connect->
rpc_clnt_add_xprt->rpc_clnt_setup_test_and_add_xprt returns error(1) but ne=
ver adds.

Thanks
-Aditya

Re: pNFS, multiple paths to Data Server not used by pNFS linux client.

[Suresh Jayaraman]

Hi Trond et al,=0A=
=0A=
On 07/19/2018 12:39 PM, Aditya Agnihotri wrote:=0A=
> =0A=
> We are testing our pNFS server with linux client, when the server returns=
 multiple paths to a v4 DS via GETDEVINFO, we see that the client validated=
 the paths (does exchange id) but does not use the alternate paths in IO.=
=0A=
> Tracing the code path, it seems that when DS is a NFS v3 server, path is =
added, but for NFS v4 DS server, client setup function callback path fails =
to add.=0A=
=0A=
Is this observation that rpc_clnt_setup_test_and_add_xprt() doesn't call=0A=
rpc_xprt_switch_add_xprt() neither in the test function nor let=0A=
rpc_clnt_add_xprt() add the xprt correct? This seems to be limiting=0A=
Linux pNFS client from using multiple paths with Tegile/WD pNFS server.=0A=
=0A=
If yes, would you like a patch that attempts to fix this?=0A=
=0A=
=0A=
--=0A=
Suresh Jayaraman=0A=

Re: pNFS, multiple paths to Data Server not used by pNFS linux client.

[Trond Myklebust]

SGkgU3VyZXNoLA0KDQpPbiBUdWUsIDIwMTgtMDctMjQgYXQgMDk6NTggKzAwMDAsIFN1cmVzaCBK
YXlhcmFtYW4gd3JvdGU6DQo+IEhpIFRyb25kIGV0IGFsLA0KPiANCj4gT24gMDcvMTkvMjAxOCAx
MjozOSBQTSwgQWRpdHlhIEFnbmlob3RyaSB3cm90ZToNCj4gPiANCj4gPiBXZSBhcmUgdGVzdGlu
ZyBvdXIgcE5GUyBzZXJ2ZXIgd2l0aCBsaW51eCBjbGllbnQsIHdoZW4gdGhlIHNlcnZlcg0KPiA+
IHJldHVybnMgbXVsdGlwbGUgcGF0aHMgdG8gYSB2NCBEUyB2aWEgR0VUREVWSU5GTywgd2Ugc2Vl
IHRoYXQgdGhlDQo+ID4gY2xpZW50IHZhbGlkYXRlZCB0aGUgcGF0aHMgKGRvZXMgZXhjaGFuZ2Ug
aWQpIGJ1dCBkb2VzIG5vdCB1c2UgdGhlDQo+ID4gYWx0ZXJuYXRlIHBhdGhzIGluIElPLg0KPiA+
IFRyYWNpbmcgdGhlIGNvZGUgcGF0aCwgaXQgc2VlbXMgdGhhdCB3aGVuIERTIGlzIGEgTkZTIHYz
IHNlcnZlciwNCj4gPiBwYXRoIGlzIGFkZGVkLCBidXQgZm9yIE5GUyB2NCBEUyBzZXJ2ZXIsIGNs
aWVudCBzZXR1cCBmdW5jdGlvbg0KPiA+IGNhbGxiYWNrIHBhdGggZmFpbHMgdG8gYWRkLg0KPiAN
Cj4gSXMgdGhpcyBvYnNlcnZhdGlvbiB0aGF0IHJwY19jbG50X3NldHVwX3Rlc3RfYW5kX2FkZF94
cHJ0KCkgZG9lc24ndA0KPiBjYWxsDQo+IHJwY194cHJ0X3N3aXRjaF9hZGRfeHBydCgpIG5laXRo
ZXIgaW4gdGhlIHRlc3QgZnVuY3Rpb24gbm9yIGxldA0KPiBycGNfY2xudF9hZGRfeHBydCgpIGFk
ZCB0aGUgeHBydCBjb3JyZWN0PyBUaGlzIHNlZW1zIHRvIGJlIGxpbWl0aW5nDQo+IExpbnV4IHBO
RlMgY2xpZW50IGZyb20gdXNpbmcgbXVsdGlwbGUgcGF0aHMgd2l0aCBUZWdpbGUvV0QgcE5GUw0K
PiBzZXJ2ZXIuDQo+IA0KPiBJZiB5ZXMsIHdvdWxkIHlvdSBsaWtlIGEgcGF0Y2ggdGhhdCBhdHRl
bXB0cyB0byBmaXggdGhpcz8NCj4gDQoNCkkgYmVsaWV2ZSB0aGF0IHRoZSBvbmx5IHJlYXNvbiB3
aHkgdGhlIE5GU3Y0IERTIHNlcnZlciBjb2RlIHdhcyBuZXZlcg0KY2hhbmdlZCBpcyB0aGF0IGl0
IHdvdWxkIG5lZWQgdG8gYmUgdGVzdGVkLCBhbmQgbm9uZSBvZiB1cyBoYXMgc3VjaCBhDQpzZXR1
cCB0byB0ZXN0IHdpdGguDQoNClNvIHllcywgaWYgeW91J3JlIGFibGUgdG8gdGVzdCBvdXQgdGhh
dCBzZXR1cCBhbmQgYXJlIHNlZWluZyBhIGJ1ZywNCnRoZW4gd2UncmUgaGFwcHkgdG8gdGFrZSBh
IGZpeC4NCg0KQ2hlZXJzDQogIFRyb25kDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZT
IGNsaWVudCBtYWludGFpbmVyLCBIYW1tZXJzcGFjZQ0KdHJvbmQubXlrbGVidXN0QGhhbW1lcnNw
YWNlLmNvbQ0KDQoNCg==

Re: pNFS, multiple paths to Data Server not used by pNFS linux client.

[Suresh Jayaraman]

VHJvbmQgTXlrbGVidXN0IDx0cm9uZG15QGhhbW1lcnNwYWNlLmNvbT4gd3JvdGU6DQoNCj4gSGkg
U3VyZXNoLA0KPiANCj4gT24gVHVlLCAyMDE4LTA3LTI0IGF0IDA5OjU4ICswMDAwLCBTdXJlc2gg
SmF5YXJhbWFuIHdyb3RlOg0KPj4gSGkgVHJvbmQgZXQgYWwsDQo+PiANCj4+IE9uIDA3LzE5LzIw
MTggMTI6MzkgUE0sIEFkaXR5YSBBZ25paG90cmkgd3JvdGU6DQo+Pj4gV2UgYXJlIHRlc3Rpbmcg
b3VyIHBORlMgc2VydmVyIHdpdGggbGludXggY2xpZW50LCB3aGVuIHRoZSBzZXJ2ZXINCj4+PiBy
ZXR1cm5zIG11bHRpcGxlIHBhdGhzIHRvIGEgdjQgRFMgdmlhIEdFVERFVklORk8sIHdlIHNlZSB0
aGF0IHRoZQ0KPj4+IGNsaWVudCB2YWxpZGF0ZWQgdGhlIHBhdGhzIChkb2VzIGV4Y2hhbmdlIGlk
KSBidXQgZG9lcyBub3QgdXNlIHRoZQ0KPj4+IGFsdGVybmF0ZSBwYXRocyBpbiBJTy4NCj4+PiBU
cmFjaW5nIHRoZSBjb2RlIHBhdGgsIGl0IHNlZW1zIHRoYXQgd2hlbiBEUyBpcyBhIE5GUyB2MyBz
ZXJ2ZXIsDQo+Pj4gcGF0aCBpcyBhZGRlZCwgYnV0IGZvciBORlMgdjQgRFMgc2VydmVyLCBjbGll
bnQgc2V0dXAgZnVuY3Rpb24NCj4+PiBjYWxsYmFjayBwYXRoIGZhaWxzIHRvIGFkZC4NCj4+IA0K
Pj4gSXMgdGhpcyBvYnNlcnZhdGlvbiB0aGF0IHJwY19jbG50X3NldHVwX3Rlc3RfYW5kX2FkZF94
cHJ0KCkgZG9lc24ndA0KPj4gY2FsbA0KPj4gcnBjX3hwcnRfc3dpdGNoX2FkZF94cHJ0KCkgbmVp
dGhlciBpbiB0aGUgdGVzdCBmdW5jdGlvbiBub3IgbGV0DQo+PiBycGNfY2xudF9hZGRfeHBydCgp
IGFkZCB0aGUgeHBydCBjb3JyZWN0PyBUaGlzIHNlZW1zIHRvIGJlIGxpbWl0aW5nDQo+PiBMaW51
eCBwTkZTIGNsaWVudCBmcm9tIHVzaW5nIG11bHRpcGxlIHBhdGhzIHdpdGggVGVnaWxlL1dEIHBO
RlMNCj4+IHNlcnZlci4NCj4+IA0KPj4gSWYgeWVzLCB3b3VsZCB5b3UgbGlrZSBhIHBhdGNoIHRo
YXQgYXR0ZW1wdHMgdG8gZml4IHRoaXM/DQo+IA0KPiBJIGJlbGlldmUgdGhhdCB0aGUgb25seSBy
ZWFzb24gd2h5IHRoZSBORlN2NCBEUyBzZXJ2ZXIgY29kZSB3YXMgbmV2ZXINCj4gY2hhbmdlZCBp
cyB0aGF0IGl0IHdvdWxkIG5lZWQgdG8gYmUgdGVzdGVkLCBhbmQgbm9uZSBvZiB1cyBoYXMgc3Vj
aCBhDQo+IHNldHVwIHRvIHRlc3Qgd2l0aC4NCj4gDQo+IFNvIHllcywgaWYgeW91J3JlIGFibGUg
dG8gdGVzdCBvdXQgdGhhdCBzZXR1cCBhbmQgYXJlIHNlZWluZyBhIGJ1ZywNCj4gdGhlbiB3ZSdy
ZSBoYXBweSB0byB0YWtlIGEgZml4Lg0KPiANCg0KVGhhbmtzIFRyb25kLiBJdCBsb29rcyBsaWtl
IHdlIG5lZWQgdG8gZ28gdGhyb3VnaCBhbiBhcHByb3ZhbCBwcm9jZXNzDQpiZWZvcmUgd2Ugc3Vi
bWl0IHRoZSBwYXRjaC4gSSBndWVzcyBpdCBzaG91bGRu4oCZdCB0YWtlIGxvbmcuDQoNClRoYW5r
cywNCg0KLS0NClN1cmVzaCBKYXlhcmFtYW4=