* Re: NFS loop on 3.4.39
[not found] <OF6F031A84.8897519B-ONC1257B4E.002D1BA3-C1257B4E.002DBED0@LocalDomain>
@ 2013-04-15 8:41 ` Joakim Tjernlund
[not found] ` <OF83CD52DB.8DBE497B-ONC1257B4E.002F8695-C1257B4E.002FBBD0@LocalDomain>
1 sibling, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-15 8:41 UTC (permalink / raw)
Cc: linux-nfs
hmm, I got another log which i a little different:
16:01:48.478917 IP 172.20.4.10.3676268197 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.479352 IP 192.168.201.44.nfs > 172.20.4.10.3676268197: reply ok
52 getattr ERROR: unk 10025
16:01:48.479394 IP 172.20.4.10.3693045413 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.479761 IP 192.168.201.44.nfs > 172.20.4.10.3693045413: reply ok
52 getattr ERROR: unk 10025
16:01:48.479887 IP 172.20.4.10.3709822629 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.480316 IP 192.168.201.44.nfs > 172.20.4.10.3709822629: reply ok
52 getattr ERROR: unk 10025
16:01:48.480415 IP 172.20.4.10.3726599845 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.480760 IP 192.168.201.44.nfs > 172.20.4.10.3726599845: reply ok
52 getattr ERROR: unk 10025
16:01:48.480882 IP 172.20.4.10.3743377061 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.481554 IP 192.168.201.44.nfs > 172.20.4.10.3743377061: reply ok
52 getattr ERROR: unk 10025
16:01:48.481652 IP 172.20.4.10.3760154277 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.482002 IP 192.168.201.44.nfs > 172.20.4.10.3760154277: reply ok
52 getattr ERROR: unk 10025
16:01:48.482085 IP 172.20.4.10.3776931493 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.482452 IP 192.168.201.44.nfs > 172.20.4.10.3776931493: reply ok
52 getattr ERROR: unk 10025
16:01:48.482537 IP 172.20.4.10.3793708709 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.482937 IP 192.168.201.44.nfs > 172.20.4.10.3793708709: reply ok
52 getattr ERROR: unk 10025
16:01:48.483050 IP 172.20.4.10.3810485925 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.483399 IP 192.168.201.44.nfs > 172.20.4.10.3810485925: reply ok
52 getattr ERROR: unk 10025
16:01:48.483439 IP 172.20.4.10.3827263141 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
16:01:48.483803 IP 192.168.201.44.nfs > 172.20.4.10.3827263141: reply ok
52 getattr ERROR: unk 10025
This is with a somewhat older kernel though: 3.4.35
The NFS sever is running 3.4.39 in both cases.
Joakim Tjernlund/Transmode wrote on 2013/04/15 10:19:39:
> From: Joakim Tjernlund/Transmode
> To: linux-nfs@vger.kernel.org,
> Date: 2013/04/15 10:19
> Subject: NFS loop on 3.4.39
>
> I get som NFS loop which generates lots of (from tcpdump):
> 9:48:43.156252 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8406616, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.156258 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8426888, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.156669 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8445712, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.156675 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8459880, win 32883, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.156679 IP 192.168.201.44.nfs > 172.20.4.10.2889942938: reply ok
52
> getattr ERROR: unk 10024
> 09:48:43.156704 IP 172.20.4.10.2923497370 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 09:48:43.156712 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8462776:8467120, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156719 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8467120:8471464, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156725 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8471464:8475808, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156731 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8475808:8480152, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156745 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8480152:8484496, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156748 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8484496:8487392, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 2896
> 09:48:43.156751 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8487392:8491736, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156755 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8491736:8496080, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156758 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8496080:8500424, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156764 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8500424:8504768, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156767 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8504768:8509112, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156770 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8509112:8512008, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 2896
> 09:48:43.156773 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8512008:8516352, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156778 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8516352:8520696, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156956 IP 192.168.201.44.nfs > 172.20.4.10.2906720154: reply ok
52
> getattr ERROR: unk 10024
> 09:48:43.156969 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8520696:8525040, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156975 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8525040:8529384, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156980 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8529384:8533728, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156984 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8533728:8536624, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 2896
> 09:48:43.156988 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8536624:8540968, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156994 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8540968:8545312, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.156999 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8545312:8549656, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.157209 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8464224, win 32872, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.157219 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8549656:8554000, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.157225 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8554000:8558344, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.157230 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [P.], seq
> 8558344:8558424, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 80
> 09:48:43.157235 IP 172.20.4.10.2940274586 > 192.168.201.44.nfs: 532
getattr fh 0,0/22
> 09:48:43.157460 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8485944, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.157709 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8504768, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.157715 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8526488, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.158126 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8543864, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.158133 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> 8558960, win 32883, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> 09:48:43.158138 IP 192.168.201.44.nfs > 172.20.4.10.2923497370: reply ok
52
> getattr ERROR: unk 10024
> 09:48:43.158143 IP 192.168.201.44.nfs > 172.20.4.10.2940274586: reply ok
52
> getattr ERROR: unk 10024
> 09:48:43.158149 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], ack
9745,
> win 23907, options [nop,nop,TS val 57419706 ecr 43795665], length 0
> 09:48:43.158174 IP 172.20.4.10.2957051802 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 09:48:43.158192 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8561856:8566200, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.158197 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8566200:8570544, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.158204 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8570544:8574888, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.158210 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8574888:8579232, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.158216 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8579232:8583576, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.158222 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8583576:8586472, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 2896
> 09:48:43.158226 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8586472:8590816, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.158233 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8590816:8595160, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
> 09:48:43.158239 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
> 8595160:8599504, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> 43795665], length 4344
>
> I wonder if this is a variant on:
>
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/
>
fs/nfs/nfs4proc.c?h=linux-3.6.y&id=9fa2b82e5592a7aa7a63b7f6a32c5aa9e580643a
>
> which does seem to be in the 3.4 stable branch?
>
> Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
[not found] ` <OF83CD52DB.8DBE497B-ONC1257B4E.002F8695-C1257B4E.002FBBD0@LocalDomain>
@ 2013-04-15 8:48 ` Joakim Tjernlund
[not found] ` <OF64305930.0F7881A0-ONC1257B4E.0030250F-C1257B4E.00306CEC@LocalDomain>
1 sibling, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-15 8:48 UTC (permalink / raw)
Cc: linux-nfs
Just as I had sent this it happened again, this time the log is different:
10:44:04.620830 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], seq
1201909:1207701, ack 4573720, win 32885, options [nop,nop,TS val 44127811
ecr 585513], length 5792
10:44:04.620836 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
1207701, win 5928, options [nop,nop,TS val 585514 ecr 44127811], length 0
10:44:04.620868 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [P.], seq
1207701:1209133, ack 4573720, win 32885, options [nop,nop,TS val 44127811
ecr 585513], length 1432
10:44:04.620899 IP 172.20.4.10.1635289962 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:44:04.620908 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
4576616:4578056, ack 1209133, win 5928, options [nop,nop,TS val 585514 ecr
44127811], length 1440
10:44:04.621365 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4578056, win 32872, options [nop,nop,TS val 44127811 ecr 585514], length 0
10:44:04.621370 IP 192.168.201.44.nfs > 172.20.4.10.1635289962: reply ok
132 getattr NON 3 ids 0/38 sz 0
10:44:04.621389 IP 172.20.4.10.1652067178 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
10:44:04.621747 IP 192.168.201.44.nfs > 172.20.4.10.1652067178: reply ok
132 getattr NON 3 ids 0/4 sz 0
10:44:04.621807 IP 172.20.4.10.1668844394 > 192.168.201.44.nfs: 236
getattr fh 0,0/22
10:44:04.621830 IP 172.20.4.10.1685621610 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
10:44:04.622263 IP 192.168.201.44.nfs > 172.20.4.10.1668844394: reply ok
136 getattr NON 3 ids 0/28 sz 0
10:44:04.622320 IP 192.168.201.44.nfs > 172.20.4.10.1685621610: reply ok
236 getattr NON 4 ids 0/15 sz 0
10:44:04.622327 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
1209785, win 5928, options [nop,nop,TS val 585514 ecr 44127811], length 0
10:44:04.622366 IP 172.20.4.10.1702398826 > 192.168.201.44.nfs: 228
getattr fh 0,0/22
10:44:04.622770 IP 192.168.201.44.nfs > 172.20.4.10.1702398826: reply ok
216 getattr NON 3 ids 0/28 sz 0
10:44:04.623288 IP 172.20.4.10.1719176042 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
10:44:04.623786 IP 192.168.201.44.nfs > 172.20.4.10.1719176042: reply ok
372 getattr NON 7 ids 0/32 sz 0
10:44:04.623841 IP 172.20.4.10.1735953258 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
10:44:04.624323 IP 192.168.201.44.nfs > 172.20.4.10.1735953258: reply ok
208 getattr NON 3 ids 0/34 sz 0
10:44:04.624386 IP 172.20.4.10.1752730474 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:44:04.624397 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
4582460:4586804, ack 1210593, win 5928, options [nop,nop,TS val 585514 ecr
44127811], length 4344
10:44:04.624404 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
4586804:4588252, ack 1210593, win 5928, options [nop,nop,TS val 585514 ecr
44127811], length 1448
10:44:04.624410 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
4588252:4588412, ack 1210593, win 5928, options [nop,nop,TS val 585514 ecr
44127811], length 160
10:44:04.624976 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4586804, win 32861, options [nop,nop,TS val 44127811 ecr 585514], length 0
10:44:04.624987 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4588412, win 32875, options [nop,nop,TS val 44127811 ecr 585514], length 0
10:44:04.625038 IP 192.168.201.44.nfs > 172.20.4.10.1752730474: reply ok
132 getattr NON 3 ids 0/38 sz 0
10:44:04.625070 IP 172.20.4.10.1769507690 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
10:44:04.625432 IP 192.168.201.44.nfs > 172.20.4.10.1769507690: reply ok
132 getattr NON 3 ids 0/4 sz 0
10:44:04.625536 IP 172.20.4.10.1786284906 > 192.168.201.44.nfs: 328
getattr fh 0,0/22
10:44:04.626082 IP 192.168.201.44.nfs > 172.20.4.10.1786284906: reply ok
372 getattr NON 7 ids 0/32 sz 0
10:44:04.626171 IP 172.20.4.10.1803062122 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
10:44:04.626672 IP 192.168.201.44.nfs > 172.20.4.10.1803062122: reply ok
208 getattr NON 3 ids 0/34 sz 0
10:44:04.626790 IP 172.20.4.10.1819839338 > 192.168.201.44.nfs: 200
getattr fh 0,0/22
10:44:04.627315 IP 192.168.201.44.nfs > 172.20.4.10.1819839338: reply ok
116 getattr NON 2 ids 0/9 sz 0
10:44:04.627407 IP 172.20.4.10.1836616554 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
10:44:04.627910 IP 192.168.201.44.nfs > 172.20.4.10.1836616554: reply ok
52 getattr ERROR: No such file or directory
10:44:04.627999 IP 172.20.4.10.1853393770 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:44:04.628009 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628015 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628021 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628024 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628030 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628102 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628112 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628147 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.628604 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4604144, win 32825, options [nop,nop,TS val 44127812 ecr 585515], length 0
10:44:04.628994 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4614280, win 32855, options [nop,nop,TS val 44127812 ecr 585515], length 0
10:44:04.629143 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4622672, win 32856, options [nop,nop,TS val 44127812 ecr 585515], length 0
10:44:04.629153 IP 192.168.201.44.nfs > 172.20.4.10.1853393770: reply ok
132 getattr NON 3 ids 0/38 sz 0
10:44:04.629207 IP 172.20.4.10.1870170986 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
10:44:04.629767 IP 192.168.201.44.nfs > 172.20.4.10.1870170986: reply ok
132 getattr NON 3 ids 0/4 sz 0
10:44:04.629898 IP 172.20.4.10.1886948202 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
10:44:04.630472 IP 192.168.201.54.1533 > 172.20.4.10.40942: Flags [P.],
seq 39752:40773, ack 1, win 353, options [nop,nop,TS val 647820778 ecr
585507], length 1021
10:44:04.630482 IP 192.168.201.44.nfs > 172.20.4.10.1886948202: reply ok
404 getattr NON 7 ids 0/32 sz 0
10:44:04.630574 IP 172.20.4.10.40942 > 192.168.201.54.1533: Flags [.], ack
40773, win 369, options [nop,nop,TS val 585516 ecr 647820778], length 0
10:44:04.630582 IP 172.20.4.10.1903725418 > 192.168.201.44.nfs: 208
getattr fh 0,0/22
10:44:04.631037 IP 192.168.201.44.nfs > 172.20.4.10.1903725418: reply ok
124 getattr NON 3 ids 0/3 sz 0
10:44:04.631130 IP 172.20.4.10.1920502634 > 192.168.201.44.nfs: 296
getattr fh 0,0/22
10:44:04.631671 IP 192.168.201.44.nfs > 172.20.4.10.1920502634: reply ok
372 getattr NON 7 ids 0/32 sz 0
10:44:04.631757 IP 172.20.4.10.1937279850 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
10:44:04.632133 IP 192.168.201.44.nfs > 172.20.4.10.1937279850: reply ok
236 getattr NON 4 ids 0/15 sz 0
10:44:04.632225 IP 172.20.4.10.1954057066 > 192.168.201.44.nfs: 240
getattr fh 0,0/22
10:44:04.632661 IP 192.168.201.44.nfs > 172.20.4.10.1954057066: reply ok
52 getattr ERROR: No such file or directory
10:44:04.632746 IP 172.20.4.10.1970834282 > 192.168.201.44.nfs: 304
getattr fh 0,0/22
10:44:04.633220 IP 192.168.201.44.nfs > 172.20.4.10.1970834282: reply ok
404 getattr NON 7 ids 0/32 sz 0
10:44:04.633313 IP 172.20.4.10.1987611498 > 192.168.201.44.nfs: 288
getattr fh 0,0/22
10:44:04.633836 IP 192.168.201.44.nfs > 172.20.4.10.1987611498: reply ok
404 getattr NON 7 ids 0/32 sz 0
10:44:04.633930 IP 172.20.4.10.2004388714 > 192.168.201.44.nfs: 200
getattr fh 0,0/22
10:44:04.634466 IP 192.168.201.44.nfs > 172.20.4.10.2004388714: reply ok
188 getattr NON 2 ids 0/9 sz 0
10:44:04.634595 IP 172.20.4.10.2021165930 > 192.168.201.44.nfs: 216
getattr fh 0,0/22
10:44:04.635213 IP 192.168.201.44.nfs > 172.20.4.10.2021165930: reply ok
4340 getattr NON 2 ids 0/25 sz 0
10:44:04.635225 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
1218461, win 5928, options [nop,nop,TS val 585517 ecr 44127812], length 0
10:44:04.635260 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [P.], seq
1218461:1222789, ack 4625228, win 32885, options [nop,nop,TS val 44127812
ecr 585517], length 4328
10:44:04.635274 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
1222789, win 5928, options [nop,nop,TS val 585517 ecr 44127812], length 0
10:44:04.635319 IP 172.20.4.10.2037943146 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:44:04.635327 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
4628124:4629564, ack 1222789, win 5928, options [nop,nop,TS val 585517 ecr
44127812], length 1440
10:44:04.636099 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4629564, win 32872, options [nop,nop,TS val 44127812 ecr 585517], length 0
10:44:04.636105 IP 192.168.201.44.nfs > 172.20.4.10.2037943146: reply ok
132 getattr NON 3 ids 0/38 sz 0
10:44:04.636135 IP 172.20.4.10.2054720362 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
10:44:04.636645 IP 192.168.201.44.nfs > 172.20.4.10.2054720362: reply ok
132 getattr NON 3 ids 0/4 sz 0
10:44:04.636698 IP 172.20.4.10.2071497578 > 192.168.201.44.nfs: 236
getattr fh 0,0/22
10:44:04.636720 IP 172.20.4.10.2088274794 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
10:44:04.637307 IP 192.168.201.44.nfs > 172.20.4.10.2071497578: reply ok
136 getattr NON 3 ids 0/28 sz 0
10:44:04.637319 IP 192.168.201.44.nfs > 172.20.4.10.2088274794: reply ok
236 getattr NON 4 ids 0/15 sz 0
10:44:04.637328 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
1223441, win 5928, options [nop,nop,TS val 585518 ecr 44127812], length 0
10:44:04.637394 IP 172.20.4.10.2105052010 > 192.168.201.44.nfs: 228
getattr fh 0,0/22
10:44:04.638006 IP 192.168.201.44.nfs > 172.20.4.10.2105052010: reply ok
216 getattr NON 3 ids 0/28 sz 0
10:44:04.638594 IP 172.20.4.10.2121829226 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
10:44:04.639181 IP 192.168.201.44.nfs > 172.20.4.10.2121829226: reply ok
372 getattr NON 7 ids 0/32 sz 0
10:44:04.639293 IP 172.20.4.10.2138606442 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
10:44:04.639737 IP 192.168.201.44.nfs > 172.20.4.10.2138606442: reply ok
208 getattr NON 3 ids 0/34 sz 0
10:44:04.639833 IP 172.20.4.10.2155383658 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:44:04.639842 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
4633968:4638312, ack 1224249, win 5928, options [nop,nop,TS val 585518 ecr
44127813], length 4344
10:44:04.639849 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
4638312:4639760, ack 1224249, win 5928, options [nop,nop,TS val 585518 ecr
44127813], length 1448
10:44:04.639855 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
4639760:4639920, ack 1224249, win 5928, options [nop,nop,TS val 585518 ecr
44127813], length 160
10:44:04.640300 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4633968, win 32878, options [nop,nop,TS val 44127813 ecr 585518], length 0
10:44:04.640387 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4639920, win 32885, options [nop,nop,TS val 44127813 ecr 585518], length 0
10:44:04.640393 IP 192.168.201.44.nfs > 172.20.4.10.2155383658: reply ok
132 getattr NON 3 ids 0/38 sz 0
10:44:04.640430 IP 172.20.4.10.2172160874 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
10:44:04.640807 IP 192.168.201.44.nfs > 172.20.4.10.2172160874: reply ok
132 getattr NON 3 ids 0/4 sz 0
10:44:04.640902 IP 172.20.4.10.2188938090 > 192.168.201.44.nfs: 208
getattr fh 0,0/22
10:44:04.641345 IP 192.168.201.44.nfs > 172.20.4.10.2188938090: reply ok
124 getattr NON 3 ids 0/3 sz 0
10:44:04.641436 IP 172.20.4.10.2205715306 > 192.168.201.44.nfs: 328
getattr fh 0,0/22
10:44:04.642009 IP 192.168.201.44.nfs > 172.20.4.10.2205715306: reply ok
372 getattr NON 7 ids 0/32 sz 0
10:44:04.642087 IP 172.20.4.10.2222492522 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
10:44:04.642601 IP 192.168.201.44.nfs > 172.20.4.10.2222492522: reply ok
208 getattr NON 3 ids 0/34 sz 0
10:44:04.642686 IP 172.20.4.10.2239269738 > 192.168.201.44.nfs: 200
getattr fh 0,0/22
10:44:04.643176 IP 192.168.201.44.nfs > 172.20.4.10.2239269738: reply ok
116 getattr NON 2 ids 0/9 sz 0
10:44:04.643209 IP 172.20.4.10.2256046954 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
10:44:04.643671 IP 192.168.201.44.nfs > 172.20.4.10.2256046954: reply ok
52 getattr ERROR: No such file or directory
10:44:04.643711 IP 172.20.4.10.2272824170 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:44:04.643722 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.643737 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.643740 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.643745 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.643750 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.643812 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.643818 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.643861 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:44:04.644229 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4650072, win 32848, options [nop,nop,TS val 44127813 ecr 585519], length 0
10:44:04.644285 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4657312, win 32861, options [nop,nop,TS val 44127813 ecr 585519], length 0
10:44:04.644526 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
4674392, win 32885, options [nop,nop,TS val 44127813 ecr 585519], length 0
10:44:04.644583 IP 192.168.201.44.nfs > 172.20.4.10.2272824170: reply ok
132 getattr NON 3 ids 0/38 sz 0
10:44:04.644641 IP 172.20.4.10.2289601386 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
10:44:04.645057 IP 192.168.201.44.nfs > 172.20.4.10.2289601386: reply ok
132 getattr NON 3 ids 0/4 sz 0
10:44:04.645149 IP 172.20.4.10.2306378602 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
10:44:04.645601 IP 192.168.201.44.nfs > 172.20.4.10.2306378602: reply ok
404 getattr NON 7 ids 0/32 sz 0
10:44:04.645661 IP 172.20.4.10.2323155818 > 192.168.201.44.nfs: 296
getattr fh 0,0/22
10:44:04.646057 IP 192.168.201.44.nfs > 172.20.4.10.2323155818: reply ok
372 getattr NON 7 ids 0/32 sz 0
10:44:04.646144 IP 172.20.4.10.2339933034 > 192.168.201.44.nfs: 240
getattr fh 0,0/22
10:44:04.646572 IP 192.168.201.44.nfs > 172.20.4.10.2339933034: reply ok
52 getattr ERROR: No such file or directory
10:44:04.646601 IP 172.20.4.10.2356710250 > 192.168.201.44.nfs: 304
getattr fh 0,0/22
10:44:04.647090 IP 192.168.201.44.nfs > 172.20.4.10.2356710250: reply ok
404 getattr NON 7 ids 0/32 sz 0
10:44:04.648915 IP 192.168.201.44.nfs > 172.20.4.10.2407041898: reply ok
2892
This time it seems like
3014 jocke 20 0 8488 1988 1636 D 14 0.0 2:28.67
gvfsd-metadata
is the victim.
Jocke
Joakim Tjernlund/Transmode wrote on 2013/04/15 10:41:22:
> From: Joakim Tjernlund/Transmode
> To:
> Cc: linux-nfs@vger.kernel.org
> Date: 2013/04/15 10:41
> Subject: Re: NFS loop on 3.4.39
>
> hmm, I got another log which i a little different:
>
> 16:01:48.478917 IP 172.20.4.10.3676268197 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.479352 IP 192.168.201.44.nfs > 172.20.4.10.3676268197: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.479394 IP 172.20.4.10.3693045413 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.479761 IP 192.168.201.44.nfs > 172.20.4.10.3693045413: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.479887 IP 172.20.4.10.3709822629 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.480316 IP 192.168.201.44.nfs > 172.20.4.10.3709822629: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.480415 IP 172.20.4.10.3726599845 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.480760 IP 192.168.201.44.nfs > 172.20.4.10.3726599845: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.480882 IP 172.20.4.10.3743377061 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.481554 IP 192.168.201.44.nfs > 172.20.4.10.3743377061: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.481652 IP 172.20.4.10.3760154277 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.482002 IP 192.168.201.44.nfs > 172.20.4.10.3760154277: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.482085 IP 172.20.4.10.3776931493 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.482452 IP 192.168.201.44.nfs > 172.20.4.10.3776931493: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.482537 IP 172.20.4.10.3793708709 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.482937 IP 192.168.201.44.nfs > 172.20.4.10.3793708709: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.483050 IP 172.20.4.10.3810485925 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.483399 IP 192.168.201.44.nfs > 172.20.4.10.3810485925: reply ok
52
> getattr ERROR: unk 10025
> 16:01:48.483439 IP 172.20.4.10.3827263141 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> 16:01:48.483803 IP 192.168.201.44.nfs > 172.20.4.10.3827263141: reply ok
52
> getattr ERROR: unk 10025
>
> This is with a somewhat older kernel though: 3.4.35
>
> The NFS sever is running 3.4.39 in both cases.
>
> Joakim Tjernlund/Transmode wrote on 2013/04/15 10:19:39:
>
> > From: Joakim Tjernlund/Transmode
> > To: linux-nfs@vger.kernel.org,
> > Date: 2013/04/15 10:19
> > Subject: NFS loop on 3.4.39
> >
> > I get som NFS loop which generates lots of (from tcpdump):
> > 9:48:43.156252 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
> > 8406616, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.156258 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8426888, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.156669 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8445712, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.156675 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8459880, win 32883, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.156679 IP 192.168.201.44.nfs > 172.20.4.10.2889942938: reply
ok 52
> > getattr ERROR: unk 10024
> > 09:48:43.156704 IP 172.20.4.10.2923497370 > 192.168.201.44.nfs: 2892
getattrfh 0,0/22
> > 09:48:43.156712 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8462776:8467120, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156719 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8467120:8471464, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156725 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8471464:8475808, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156731 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8475808:8480152, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156745 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8480152:8484496, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156748 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8484496:8487392, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 2896
> > 09:48:43.156751 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8487392:8491736, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156755 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8491736:8496080, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156758 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8496080:8500424, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156764 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8500424:8504768, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156767 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8504768:8509112, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156770 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8509112:8512008, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 2896
> > 09:48:43.156773 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8512008:8516352, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156778 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8516352:8520696, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156956 IP 192.168.201.44.nfs > 172.20.4.10.2906720154: reply
ok 52
> > getattr ERROR: unk 10024
> > 09:48:43.156969 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8520696:8525040, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156975 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8525040:8529384, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156980 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8529384:8533728, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156984 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8533728:8536624, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 2896
> > 09:48:43.156988 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8536624:8540968, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156994 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8540968:8545312, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.156999 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8545312:8549656, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.157209 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8464224, win 32872, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.157219 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8549656:8554000, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.157225 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8554000:8558344, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.157230 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [P.],
seq
> > 8558344:8558424, ack 9577, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 80
> > 09:48:43.157235 IP 172.20.4.10.2940274586 > 192.168.201.44.nfs: 532
getattr fh 0,0/22
> > 09:48:43.157460 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8485944, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.157709 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8504768, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.157715 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8526488, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.158126 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8543864, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.158133 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > 8558960, win 32883, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > 09:48:43.158138 IP 192.168.201.44.nfs > 172.20.4.10.2923497370: reply
ok 52
> > getattr ERROR: unk 10024
> > 09:48:43.158143 IP 192.168.201.44.nfs > 172.20.4.10.2940274586: reply
ok 52
> > getattr ERROR: unk 10024
> > 09:48:43.158149 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
ack 9745,
> > win 23907, options [nop,nop,TS val 57419706 ecr 43795665], length 0
> > 09:48:43.158174 IP 172.20.4.10.2957051802 > 192.168.201.44.nfs: 2892
getattrfh 0,0/22
> > 09:48:43.158192 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8561856:8566200, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.158197 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8566200:8570544, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.158204 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8570544:8574888, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.158210 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8574888:8579232, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.158216 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8579232:8583576, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.158222 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8583576:8586472, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 2896
> > 09:48:43.158226 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8586472:8590816, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.158233 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8590816:8595160, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> > 09:48:43.158239 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > 8595160:8599504, ack 9745, win 23907, options [nop,nop,TS val 57419706
ecr
> > 43795665], length 4344
> >
> > I wonder if this is a variant on:
> >
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/
> >
fs/nfs/nfs4proc.c?h=linux-3.6.y&id=9fa2b82e5592a7aa7a63b7f6a32c5aa9e580643a
> >
> > which does seem to be in the 3.4 stable branch?
> >
> > Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
[not found] ` <OF64305930.0F7881A0-ONC1257B4E.0030250F-C1257B4E.00306CEC@LocalDomain>
@ 2013-04-16 10:41 ` Joakim Tjernlund
2013-04-16 15:36 ` Myklebust, Trond
0 siblings, 1 reply; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-16 10:41 UTC (permalink / raw)
Cc: linux-nfs
Here we go again, this time i happened while browsing the Boston news on
www.dn.se
Now gvfsd-metadata is turned off(not running at all) and I get:
10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838: reply ok
52 getattr ERROR: unk 10024
10:28:44.616170 IP 172.20.4.10.3688546054 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:28:44.616181 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616188 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616195 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616202 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616220 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616226 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616231 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616235 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616238 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113785365:113789709, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616242 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113789709:113794053, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616248 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113794053:113798397, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616251 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113798397:113801293, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 2896
10:28:44.616256 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113801293:113805637, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616264 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113805637:113809981, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616269 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113809981:113814325, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616272 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113814325:113818669, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616276 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616314 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113823013:113825909, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 2896
10:28:44.616317 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113825909:113830253, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616372 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113830253:113834597, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616425 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113834597:113838941, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616432 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113838941:113843285, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616469 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113843285:113846181, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 2896
10:28:44.616523 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113846181:113850525, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616583 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113752061, win 32878, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.616597 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113850525:113854869, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616600 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616617 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113859213:113863557, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616667 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113863557:113867901, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616721 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113867901:113870797, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 2896
10:28:44.616766 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113767989, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.616776 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113870797:113875141, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616783 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113875141:113879485, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616818 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113772333, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.616824 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113879485:113883829, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616871 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113883829:113888173, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.616926 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.616933 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113892517:113895413, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 2896
10:28:44.617018 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113895413:113899757, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617024 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113899757:113904101, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617069 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113785365, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.617078 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113904101:113908445, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617085 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113908445:113912789, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617114 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113912789:113917133, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617169 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113917133:113920029, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 2896
10:28:44.617221 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113808533, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.617229 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.617269 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113924373:113928717, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617275 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113928717:113933061, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617314 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113933061:113937405, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617372 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113937405:113941749, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 4344
10:28:44.617415 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113831701, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.617424 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113941749:113944645, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 2896
10:28:44.617431 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113944645:113946093, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 1448
10:28:44.617472 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [P.], seq
113946093:113946157, ack 33936, win 24576, options [nop,nop,TS val
19887700 ecr 52675811], length 64
10:28:44.617567 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113851973, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.617770 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113865005, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.617965 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113882381, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.618184 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113904101, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.618432 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113927269, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.618547 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
113946157, win 32885, options [nop,nop,TS val 52675811 ecr 19887700],
length 0
10:28:44.618553 IP 192.168.201.44.nfs > 172.20.4.10.3688546054: reply ok
52 getattr ERROR: unk 10024
10:28:44.618574 IP 172.20.4.10.3705323270 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:28:44.618590 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618594 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618602 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618607 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618612 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618617 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618622 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618625 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.618629 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113982357:113986701, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 4344
10:28:44.618634 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113986701:113991045, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 4344
10:28:44.618638 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113991045:113995389, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 4344
10:28:44.618642 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113995389:113998285, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 2896
10:28:44.618646 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
113998285:114002629, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 4344
10:28:44.618656 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
114002629:114006973, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 4344
10:28:44.618662 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
114006973:114011317, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 4344
10:28:44.618665 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
114011317:114015661, ack 33992, win 24576, options [nop,nop,TS val
19887701 ecr 52675811], length 4344
10:28:44.636565 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
115255533, win 32885, options [nop,nop,TS val 52675813 ecr 19887705],
length 0
10:28:44.644779 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
115865333, win 32885, options [nop,nop,TS val 52675813 ecr 19887707],
length 0
10:28:44.652558 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
116404181, win 32885, options [nop,nop,TS val 52675814 ecr 19887709],
length 0
10:28:44.652774 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
116427349, win 32885, options [nop,nop,TS val 52675814 ecr 19887709],
length 0
10:28:44.653089 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
116449069, win 32885, options [nop,nop,TS val 52675814 ecr 19887709],
length 0
10:28:44.653225 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
116470789, win 32885, options [nop,nop,TS val 52675814 ecr 19887709],
length 0
10:28:44.653378 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
116483821, win 32885, options [nop,nop,TS val 52675814 ecr 19887709],
length 0
10:28:44.653525 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
116499749, win 32885, options [nop,nop,TS val 52675814 ecr 19887709],
length 0
10:28:44.653793 IP 192.168.201.44.nfs > 172.20.4.10.725: Flags [.], ack
116507053, win 32885, options [nop,nop,TS val 52675814 ecr 19887709],
length 0
10:28:44.653799 IP 192.168.201.44.nfs > 172.20.4.10.3906649862: reply ok
52
10:28:44.653825 IP 172.20.4.10.3923427078 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
10:28:44.653836 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653844 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653851 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653865 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653870 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653875 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653881 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653884 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653888 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116543253:116547597, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.653891 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116547597:116551941, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.653894 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116551941:116556285, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.653898 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116556285:116559181, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 2896
10:28:44.653902 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116559181:116563525, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.653908 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116563525:116567869, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.653912 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116567869:116572213, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.653921 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116572213:116576557, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.653926 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
10:28:44.653971 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116580901:116583797, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 2896
10:28:44.653974 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116583797:116588141, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.654020 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116588141:116592485, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.654072 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116592485:116596829, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.654078 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116596829:116601173, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 4344
10:28:44.654125 IP 172.20.4.10.725 > 192.168.201.44.nfs: Flags [.], seq
116601173:116604069, ack 34720, win 24576, options [nop,nop,TS val
19887710 ecr 52675814], length 2896
This pattern pretty much repeats. Firefox has hung up too:
ps uaxww| grep firefox
jocke 6330 0.2 0.0 0 0 tty1 Zl Apr15 3:04 [firefox]
<defunct>
jocke 10655 0.0 6.8 733648 279432 tty1 D 10:25 0:00 firefox
jocke 10691 0.0 6.8 733648 279432 tty1 D 10:26 0:00 firefox
restarting client nfs(on gentoo) /etc/init.d/nfs restart
does nothing.
Anyone?
Jocke
Joakim Tjernlund/Transmode wrote on 2013/04/15 10:48:56:
> From: Joakim Tjernlund/Transmode
> To:
> Cc: linux-nfs@vger.kernel.org
> Date: 2013/04/15 10:48
> Subject: Re: NFS loop on 3.4.39
>
> Just as I had sent this it happened again, this time the log is
different:
>
> 10:44:04.620830 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], seq
> 1201909:1207701, ack 4573720, win 32885, options [nop,nop,TS val
44127811 ecr
> 585513], length 5792
> 10:44:04.620836 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
> 1207701, win 5928, options [nop,nop,TS val 585514 ecr 44127811], length
0
> 10:44:04.620868 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [P.], seq
> 1207701:1209133, ack 4573720, win 32885, options [nop,nop,TS val
44127811 ecr
> 585513], length 1432
> 10:44:04.620899 IP 172.20.4.10.1635289962 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 10:44:04.620908 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
> 4576616:4578056, ack 1209133, win 5928, options [nop,nop,TS val 585514
ecr
> 44127811], length 1440
> 10:44:04.621365 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4578056, win 32872, options [nop,nop,TS val 44127811 ecr 585514], length
0
> 10:44:04.621370 IP 192.168.201.44.nfs > 172.20.4.10.1635289962: reply ok
132
> getattr NON 3 ids 0/38 sz 0
> 10:44:04.621389 IP 172.20.4.10.1652067178 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
> 10:44:04.621747 IP 192.168.201.44.nfs > 172.20.4.10.1652067178: reply ok
132
> getattr NON 3 ids 0/4 sz 0
> 10:44:04.621807 IP 172.20.4.10.1668844394 > 192.168.201.44.nfs: 236
getattr fh 0,0/22
> 10:44:04.621830 IP 172.20.4.10.1685621610 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
> 10:44:04.622263 IP 192.168.201.44.nfs > 172.20.4.10.1668844394: reply ok
136
> getattr NON 3 ids 0/28 sz 0
> 10:44:04.622320 IP 192.168.201.44.nfs > 172.20.4.10.1685621610: reply ok
236
> getattr NON 4 ids 0/15 sz 0
> 10:44:04.622327 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
> 1209785, win 5928, options [nop,nop,TS val 585514 ecr 44127811], length
0
> 10:44:04.622366 IP 172.20.4.10.1702398826 > 192.168.201.44.nfs: 228
getattr fh 0,0/22
> 10:44:04.622770 IP 192.168.201.44.nfs > 172.20.4.10.1702398826: reply ok
216
> getattr NON 3 ids 0/28 sz 0
> 10:44:04.623288 IP 172.20.4.10.1719176042 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
> 10:44:04.623786 IP 192.168.201.44.nfs > 172.20.4.10.1719176042: reply ok
372
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.623841 IP 172.20.4.10.1735953258 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
> 10:44:04.624323 IP 192.168.201.44.nfs > 172.20.4.10.1735953258: reply ok
208
> getattr NON 3 ids 0/34 sz 0
> 10:44:04.624386 IP 172.20.4.10.1752730474 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 10:44:04.624397 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
> 4582460:4586804, ack 1210593, win 5928, options [nop,nop,TS val 585514
ecr
> 44127811], length 4344
> 10:44:04.624404 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
> 4586804:4588252, ack 1210593, win 5928, options [nop,nop,TS val 585514
ecr
> 44127811], length 1448
> 10:44:04.624410 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
> 4588252:4588412, ack 1210593, win 5928, options [nop,nop,TS val 585514
ecr
> 44127811], length 160
> 10:44:04.624976 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4586804, win 32861, options [nop,nop,TS val 44127811 ecr 585514], length
0
> 10:44:04.624987 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4588412, win 32875, options [nop,nop,TS val 44127811 ecr 585514], length
0
> 10:44:04.625038 IP 192.168.201.44.nfs > 172.20.4.10.1752730474: reply ok
132
> getattr NON 3 ids 0/38 sz 0
> 10:44:04.625070 IP 172.20.4.10.1769507690 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
> 10:44:04.625432 IP 192.168.201.44.nfs > 172.20.4.10.1769507690: reply ok
132
> getattr NON 3 ids 0/4 sz 0
> 10:44:04.625536 IP 172.20.4.10.1786284906 > 192.168.201.44.nfs: 328
getattr fh 0,0/22
> 10:44:04.626082 IP 192.168.201.44.nfs > 172.20.4.10.1786284906: reply ok
372
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.626171 IP 172.20.4.10.1803062122 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
> 10:44:04.626672 IP 192.168.201.44.nfs > 172.20.4.10.1803062122: reply ok
208
> getattr NON 3 ids 0/34 sz 0
> 10:44:04.626790 IP 172.20.4.10.1819839338 > 192.168.201.44.nfs: 200
getattr fh 0,0/22
> 10:44:04.627315 IP 192.168.201.44.nfs > 172.20.4.10.1819839338: reply ok
116
> getattr NON 2 ids 0/9 sz 0
> 10:44:04.627407 IP 172.20.4.10.1836616554 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
> 10:44:04.627910 IP 192.168.201.44.nfs > 172.20.4.10.1836616554: reply ok
52
> getattr ERROR: No such file or directory
> 10:44:04.627999 IP 172.20.4.10.1853393770 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 10:44:04.628009 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628015 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628021 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628024 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628030 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628102 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628112 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628147 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.628604 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4604144, win 32825, options [nop,nop,TS val 44127812 ecr 585515], length
0
> 10:44:04.628994 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4614280, win 32855, options [nop,nop,TS val 44127812 ecr 585515], length
0
> 10:44:04.629143 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4622672, win 32856, options [nop,nop,TS val 44127812 ecr 585515], length
0
> 10:44:04.629153 IP 192.168.201.44.nfs > 172.20.4.10.1853393770: reply ok
132
> getattr NON 3 ids 0/38 sz 0
> 10:44:04.629207 IP 172.20.4.10.1870170986 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
> 10:44:04.629767 IP 192.168.201.44.nfs > 172.20.4.10.1870170986: reply ok
132
> getattr NON 3 ids 0/4 sz 0
> 10:44:04.629898 IP 172.20.4.10.1886948202 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
> 10:44:04.630472 IP 192.168.201.54.1533 > 172.20.4.10.40942: Flags [P.],
seq
> 39752:40773, ack 1, win 353, options [nop,nop,TS val 647820778 ecr
585507], length 1021
> 10:44:04.630482 IP 192.168.201.44.nfs > 172.20.4.10.1886948202: reply ok
404
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.630574 IP 172.20.4.10.40942 > 192.168.201.54.1533: Flags [.],
ack
> 40773, win 369, options [nop,nop,TS val 585516 ecr 647820778], length 0
> 10:44:04.630582 IP 172.20.4.10.1903725418 > 192.168.201.44.nfs: 208
getattr fh 0,0/22
> 10:44:04.631037 IP 192.168.201.44.nfs > 172.20.4.10.1903725418: reply ok
124
> getattr NON 3 ids 0/3 sz 0
> 10:44:04.631130 IP 172.20.4.10.1920502634 > 192.168.201.44.nfs: 296
getattr fh 0,0/22
> 10:44:04.631671 IP 192.168.201.44.nfs > 172.20.4.10.1920502634: reply ok
372
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.631757 IP 172.20.4.10.1937279850 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
> 10:44:04.632133 IP 192.168.201.44.nfs > 172.20.4.10.1937279850: reply ok
236
> getattr NON 4 ids 0/15 sz 0
> 10:44:04.632225 IP 172.20.4.10.1954057066 > 192.168.201.44.nfs: 240
getattr fh 0,0/22
> 10:44:04.632661 IP 192.168.201.44.nfs > 172.20.4.10.1954057066: reply ok
52
> getattr ERROR: No such file or directory
> 10:44:04.632746 IP 172.20.4.10.1970834282 > 192.168.201.44.nfs: 304
getattr fh 0,0/22
> 10:44:04.633220 IP 192.168.201.44.nfs > 172.20.4.10.1970834282: reply ok
404
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.633313 IP 172.20.4.10.1987611498 > 192.168.201.44.nfs: 288
getattr fh 0,0/22
> 10:44:04.633836 IP 192.168.201.44.nfs > 172.20.4.10.1987611498: reply ok
404
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.633930 IP 172.20.4.10.2004388714 > 192.168.201.44.nfs: 200
getattr fh 0,0/22
> 10:44:04.634466 IP 192.168.201.44.nfs > 172.20.4.10.2004388714: reply ok
188
> getattr NON 2 ids 0/9 sz 0
> 10:44:04.634595 IP 172.20.4.10.2021165930 > 192.168.201.44.nfs: 216
getattr fh 0,0/22
> 10:44:04.635213 IP 192.168.201.44.nfs > 172.20.4.10.2021165930: reply ok
4340
> getattr NON 2 ids 0/25 sz 0
> 10:44:04.635225 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
> 1218461, win 5928, options [nop,nop,TS val 585517 ecr 44127812], length
0
> 10:44:04.635260 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [P.], seq
> 1218461:1222789, ack 4625228, win 32885, options [nop,nop,TS val
44127812 ecr
> 585517], length 4328
> 10:44:04.635274 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
> 1222789, win 5928, options [nop,nop,TS val 585517 ecr 44127812], length
0
> 10:44:04.635319 IP 172.20.4.10.2037943146 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 10:44:04.635327 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
> 4628124:4629564, ack 1222789, win 5928, options [nop,nop,TS val 585517
ecr
> 44127812], length 1440
> 10:44:04.636099 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4629564, win 32872, options [nop,nop,TS val 44127812 ecr 585517], length
0
> 10:44:04.636105 IP 192.168.201.44.nfs > 172.20.4.10.2037943146: reply ok
132
> getattr NON 3 ids 0/38 sz 0
> 10:44:04.636135 IP 172.20.4.10.2054720362 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
> 10:44:04.636645 IP 192.168.201.44.nfs > 172.20.4.10.2054720362: reply ok
132
> getattr NON 3 ids 0/4 sz 0
> 10:44:04.636698 IP 172.20.4.10.2071497578 > 192.168.201.44.nfs: 236
getattr fh 0,0/22
> 10:44:04.636720 IP 172.20.4.10.2088274794 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
> 10:44:04.637307 IP 192.168.201.44.nfs > 172.20.4.10.2071497578: reply ok
136
> getattr NON 3 ids 0/28 sz 0
> 10:44:04.637319 IP 192.168.201.44.nfs > 172.20.4.10.2088274794: reply ok
236
> getattr NON 4 ids 0/15 sz 0
> 10:44:04.637328 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], ack
> 1223441, win 5928, options [nop,nop,TS val 585518 ecr 44127812], length
0
> 10:44:04.637394 IP 172.20.4.10.2105052010 > 192.168.201.44.nfs: 228
getattr fh 0,0/22
> 10:44:04.638006 IP 192.168.201.44.nfs > 172.20.4.10.2105052010: reply ok
216
> getattr NON 3 ids 0/28 sz 0
> 10:44:04.638594 IP 172.20.4.10.2121829226 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
> 10:44:04.639181 IP 192.168.201.44.nfs > 172.20.4.10.2121829226: reply ok
372
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.639293 IP 172.20.4.10.2138606442 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
> 10:44:04.639737 IP 192.168.201.44.nfs > 172.20.4.10.2138606442: reply ok
208
> getattr NON 3 ids 0/34 sz 0
> 10:44:04.639833 IP 172.20.4.10.2155383658 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 10:44:04.639842 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
> 4633968:4638312, ack 1224249, win 5928, options [nop,nop,TS val 585518
ecr
> 44127813], length 4344
> 10:44:04.639849 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [.], seq
> 4638312:4639760, ack 1224249, win 5928, options [nop,nop,TS val 585518
ecr
> 44127813], length 1448
> 10:44:04.639855 IP 172.20.4.10.763 > 192.168.201.44.nfs: Flags [P.], seq
> 4639760:4639920, ack 1224249, win 5928, options [nop,nop,TS val 585518
ecr
> 44127813], length 160
> 10:44:04.640300 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4633968, win 32878, options [nop,nop,TS val 44127813 ecr 585518], length
0
> 10:44:04.640387 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4639920, win 32885, options [nop,nop,TS val 44127813 ecr 585518], length
0
> 10:44:04.640393 IP 192.168.201.44.nfs > 172.20.4.10.2155383658: reply ok
132
> getattr NON 3 ids 0/38 sz 0
> 10:44:04.640430 IP 172.20.4.10.2172160874 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
> 10:44:04.640807 IP 192.168.201.44.nfs > 172.20.4.10.2172160874: reply ok
132
> getattr NON 3 ids 0/4 sz 0
> 10:44:04.640902 IP 172.20.4.10.2188938090 > 192.168.201.44.nfs: 208
getattr fh 0,0/22
> 10:44:04.641345 IP 192.168.201.44.nfs > 172.20.4.10.2188938090: reply ok
124
> getattr NON 3 ids 0/3 sz 0
> 10:44:04.641436 IP 172.20.4.10.2205715306 > 192.168.201.44.nfs: 328
getattr fh 0,0/22
> 10:44:04.642009 IP 192.168.201.44.nfs > 172.20.4.10.2205715306: reply ok
372
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.642087 IP 172.20.4.10.2222492522 > 192.168.201.44.nfs: 248
getattr fh 0,0/22
> 10:44:04.642601 IP 192.168.201.44.nfs > 172.20.4.10.2222492522: reply ok
208
> getattr NON 3 ids 0/34 sz 0
> 10:44:04.642686 IP 172.20.4.10.2239269738 > 192.168.201.44.nfs: 200
getattr fh 0,0/22
> 10:44:04.643176 IP 192.168.201.44.nfs > 172.20.4.10.2239269738: reply ok
116
> getattr NON 2 ids 0/9 sz 0
> 10:44:04.643209 IP 172.20.4.10.2256046954 > 192.168.201.44.nfs: 232
getattr fh 0,0/22
> 10:44:04.643671 IP 192.168.201.44.nfs > 172.20.4.10.2256046954: reply ok
52
> getattr ERROR: No such file or directory
> 10:44:04.643711 IP 172.20.4.10.2272824170 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
> 10:44:04.643722 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.643737 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.643740 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.643745 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.643750 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.643812 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.643818 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.643861 IP 172.20.4.10.0 > 192.168.201.44.nfs: 0 null
> 10:44:04.644229 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4650072, win 32848, options [nop,nop,TS val 44127813 ecr 585519], length
0
> 10:44:04.644285 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4657312, win 32861, options [nop,nop,TS val 44127813 ecr 585519], length
0
> 10:44:04.644526 IP 192.168.201.44.nfs > 172.20.4.10.763: Flags [.], ack
> 4674392, win 32885, options [nop,nop,TS val 44127813 ecr 585519], length
0
> 10:44:04.644583 IP 192.168.201.44.nfs > 172.20.4.10.2272824170: reply ok
132
> getattr NON 3 ids 0/38 sz 0
> 10:44:04.644641 IP 172.20.4.10.2289601386 > 192.168.201.44.nfs: 224
getattr fh 0,0/22
> 10:44:04.645057 IP 192.168.201.44.nfs > 172.20.4.10.2289601386: reply ok
132
> getattr NON 3 ids 0/4 sz 0
> 10:44:04.645149 IP 172.20.4.10.2306378602 > 192.168.201.44.nfs: 316
getattr fh 0,0/22
> 10:44:04.645601 IP 192.168.201.44.nfs > 172.20.4.10.2306378602: reply ok
404
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.645661 IP 172.20.4.10.2323155818 > 192.168.201.44.nfs: 296
getattr fh 0,0/22
> 10:44:04.646057 IP 192.168.201.44.nfs > 172.20.4.10.2323155818: reply ok
372
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.646144 IP 172.20.4.10.2339933034 > 192.168.201.44.nfs: 240
getattr fh 0,0/22
> 10:44:04.646572 IP 192.168.201.44.nfs > 172.20.4.10.2339933034: reply ok
52
> getattr ERROR: No such file or directory
> 10:44:04.646601 IP 172.20.4.10.2356710250 > 192.168.201.44.nfs: 304
getattr fh 0,0/22
> 10:44:04.647090 IP 192.168.201.44.nfs > 172.20.4.10.2356710250: reply ok
404
> getattr NON 7 ids 0/32 sz 0
> 10:44:04.648915 IP 192.168.201.44.nfs > 172.20.4.10.2407041898: reply ok
2892
>
> This time it seems like
> 3014 jocke 20 0 8488 1988 1636 D 14 0.0 2:28.67
gvfsd-metadata
> is the victim.
>
> Jocke
>
> Joakim Tjernlund/Transmode wrote on 2013/04/15 10:41:22:
>
> > From: Joakim Tjernlund/Transmode
> > To:
> > Cc: linux-nfs@vger.kernel.org
> > Date: 2013/04/15 10:41
> > Subject: Re: NFS loop on 3.4.39
> >
> > hmm, I got another log which i a little different:
> >
> > 16:01:48.478917 IP 172.20.4.10.3676268197 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.479352 IP 192.168.201.44.nfs > 172.20.4.10.3676268197: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.479394 IP 172.20.4.10.3693045413 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.479761 IP 192.168.201.44.nfs > 172.20.4.10.3693045413: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.479887 IP 172.20.4.10.3709822629 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.480316 IP 192.168.201.44.nfs > 172.20.4.10.3709822629: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.480415 IP 172.20.4.10.3726599845 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.480760 IP 192.168.201.44.nfs > 172.20.4.10.3726599845: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.480882 IP 172.20.4.10.3743377061 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.481554 IP 192.168.201.44.nfs > 172.20.4.10.3743377061: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.481652 IP 172.20.4.10.3760154277 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.482002 IP 192.168.201.44.nfs > 172.20.4.10.3760154277: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.482085 IP 172.20.4.10.3776931493 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.482452 IP 192.168.201.44.nfs > 172.20.4.10.3776931493: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.482537 IP 172.20.4.10.3793708709 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.482937 IP 192.168.201.44.nfs > 172.20.4.10.3793708709: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.483050 IP 172.20.4.10.3810485925 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.483399 IP 192.168.201.44.nfs > 172.20.4.10.3810485925: reply
ok 52
> > getattr ERROR: unk 10025
> > 16:01:48.483439 IP 172.20.4.10.3827263141 > 192.168.201.44.nfs: 272
getattr fh 0,0/22
> > 16:01:48.483803 IP 192.168.201.44.nfs > 172.20.4.10.3827263141: reply
ok 52
> > getattr ERROR: unk 10025
> >
> > This is with a somewhat older kernel though: 3.4.35
> >
> > The NFS sever is running 3.4.39 in both cases.
> >
> > Joakim Tjernlund/Transmode wrote on 2013/04/15 10:19:39:
> >
> > > From: Joakim Tjernlund/Transmode
> > > To: linux-nfs@vger.kernel.org,
> > > Date: 2013/04/15 10:19
> > > Subject: NFS loop on 3.4.39
> > >
> > > I get som NFS loop which generates lots of (from tcpdump):
> > > 9:48:43.156252 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8406616, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.156258 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8426888, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.156669 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8445712, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.156675 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8459880, win 32883, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.156679 IP 192.168.201.44.nfs > 172.20.4.10.2889942938:
reply ok 52
> > > getattr ERROR: unk 10024
> > > 09:48:43.156704 IP 172.20.4.10.2923497370 > 192.168.201.44.nfs: 2892
> getattrfh 0,0/22
> > > 09:48:43.156712 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8462776:8467120, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156719 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8467120:8471464, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156725 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8471464:8475808, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156731 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8475808:8480152, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156745 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8480152:8484496, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156748 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8484496:8487392, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 2896
> > > 09:48:43.156751 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8487392:8491736, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156755 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8491736:8496080, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156758 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8496080:8500424, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156764 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8500424:8504768, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156767 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8504768:8509112, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156770 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8509112:8512008, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 2896
> > > 09:48:43.156773 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8512008:8516352, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156778 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8516352:8520696, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156956 IP 192.168.201.44.nfs > 172.20.4.10.2906720154:
reply ok 52
> > > getattr ERROR: unk 10024
> > > 09:48:43.156969 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8520696:8525040, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156975 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8525040:8529384, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156980 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8529384:8533728, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156984 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8533728:8536624, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 2896
> > > 09:48:43.156988 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8536624:8540968, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156994 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8540968:8545312, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.156999 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8545312:8549656, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.157209 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8464224, win 32872, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.157219 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8549656:8554000, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.157225 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8554000:8558344, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.157230 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [P.],
seq
> > > 8558344:8558424, ack 9577, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 80
> > > 09:48:43.157235 IP 172.20.4.10.2940274586 > 192.168.201.44.nfs: 532
> getattr fh 0,0/22
> > > 09:48:43.157460 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8485944, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.157709 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8504768, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.157715 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8526488, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.158126 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8543864, win 32885, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.158133 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.],
ack
> > > 8558960, win 32883, options [nop,nop,TS val 43795665 ecr 57419706],
length 0
> > > 09:48:43.158138 IP 192.168.201.44.nfs > 172.20.4.10.2923497370:
reply ok 52
> > > getattr ERROR: unk 10024
> > > 09:48:43.158143 IP 192.168.201.44.nfs > 172.20.4.10.2940274586:
reply ok 52
> > > getattr ERROR: unk 10024
> > > 09:48:43.158149 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
ack 9745,
> > > win 23907, options [nop,nop,TS val 57419706 ecr 43795665], length 0
> > > 09:48:43.158174 IP 172.20.4.10.2957051802 > 192.168.201.44.nfs: 2892
> getattrfh 0,0/22
> > > 09:48:43.158192 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8561856:8566200, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.158197 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8566200:8570544, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.158204 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8570544:8574888, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.158210 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8574888:8579232, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.158216 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8579232:8583576, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.158222 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8583576:8586472, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 2896
> > > 09:48:43.158226 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8586472:8590816, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.158233 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8590816:8595160, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > > 09:48:43.158239 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.],
seq
> > > 8595160:8599504, ack 9745, win 23907, options [nop,nop,TS val
57419706 ecr
> > > 43795665], length 4344
> > >
> > > I wonder if this is a variant on:
> > >
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/
> > >
fs/nfs/nfs4proc.c?h=linux-3.6.y&id=9fa2b82e5592a7aa7a63b7f6a32c5aa9e580643a
> > >
> > > which does seem to be in the 3.4 stable branch?
> > >
> > > Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-16 10:41 ` Joakim Tjernlund
@ 2013-04-16 15:36 ` Myklebust, Trond
2013-04-16 19:07 ` Joakim Tjernlund
0 siblings, 1 reply; 21+ messages in thread
From: Myklebust, Trond @ 2013-04-16 15:36 UTC (permalink / raw)
To: Joakim Tjernlund; +Cc: linux-nfs
On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> Here we go again, this time i happened while browsing the Boston news on
> www.dn.se
> Now gvfsd-metadata is turned off(not running at all) and I get:
> 10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838: reply ok
> 52 getattr ERROR: unk 10024
Part of the reason why you are getting no response to these posts is
that you are posting tcpdump-decoded data. Tcpdump still has no support
for NFSv4, and therefore completely garbles the output by trying to
interpret it as NFSv2/v3.
In general, if you are posting network traffic, please record it as
binary raw packet data (using the '-w' option on tcdump) so that we can
look at the full contents. Either include it as an attachment, or
provide us with details on how to download it from an http server.
Other information that is needed in order to make sense of NFS bug
reports includes:
- client OS (non-linux) or kernel version (linux)
- mount options on the client
- server OS (non-linux) or kernel version (linux)
- type of exported filesystem on the server
- contents of /etc/exports on the server
Please ensure that you always include those in your emails.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@netapp.com
www.netapp.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-16 15:36 ` Myklebust, Trond
@ 2013-04-16 19:07 ` Joakim Tjernlund
2013-04-16 22:06 ` Myklebust, Trond
0 siblings, 1 reply; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-16 19:07 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/16
17:36:55:
> From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> Date: 2013/04/16 17:37
> Subject: Re: NFS loop on 3.4.39
>
> On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > Here we go again, this time i happened while browsing the Boston news
on
> > www.dn.se
> > Now gvfsd-metadata is turned off(not running at all) and I get:
> > 10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838: reply
ok
> > 52 getattr ERROR: unk 10024
>
> Part of the reason why you are getting no response to these posts is
> that you are posting tcpdump-decoded data. Tcpdump still has no support
> for NFSv4, and therefore completely garbles the output by trying to
> interpret it as NFSv2/v3.
> In general, if you are posting network traffic, please record it as
> binary raw packet data (using the '-w' option on tcdump) so that we can
> look at the full contents. Either include it as an attachment, or
> provide us with details on how to download it from an http server.
>
> Other information that is needed in order to make sense of NFS bug
> reports includes:
Thank you Trond, I figured there was something missing but I didn't know
where to start but here goes:
>
> - client OS (non-linux) or kernel version (linux)
Client OS Linux 3.4.39, x86
> - mount options on the client
~ # ypmatch jocke auto.home
-fstype=nfs,soft devsrv:/mnt/home/jocke
> - server OS (non-linux) or kernel version (linux)
Server OS Linux 3.4.39, amd64
> - type of exported filesystem on the server
XFS
> - contents of /etc/exports on the server
more /etc/exports
# /etc/exports: NFS file systems being exported. See exports(5).
/mnt/home *(rw,async,root_squash,no_subtree_check)
/mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
/mnt/TNM *(rw,sync,root_squash,no_subtree_check)
/tftproot *(rw,async,root_squash,no_subtree_check)
/mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
/rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
/mnt/home is the one failing
>
> Please ensure that you always include those in your emails.
nfs.pcap:
http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
nfs2.pcap:
http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
nfs3.pcap:
http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
nfs4.pcap:
http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
nfs3.pcap is the gvsd-metadata problem one can find using google, doesn't
have to be a NFS problem
The other 3 all come from surfing the www using firefox 17.0.3
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-16 19:07 ` Joakim Tjernlund
@ 2013-04-16 22:06 ` Myklebust, Trond
2013-04-16 22:43 ` Joakim Tjernlund
` (3 more replies)
0 siblings, 4 replies; 21+ messages in thread
From: Myklebust, Trond @ 2013-04-16 22:06 UTC (permalink / raw)
To: Joakim Tjernlund; +Cc: linux-nfs
On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/16
> 17:36:55:
>
> > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > Date: 2013/04/16 17:37
> > Subject: Re: NFS loop on 3.4.39
> >
> > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > Here we go again, this time i happened while browsing the Boston news
> on
> > > www.dn.se
> > > Now gvfsd-metadata is turned off(not running at all) and I get:
> > > 10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838: reply
> ok
> > > 52 getattr ERROR: unk 10024
> >
> > Part of the reason why you are getting no response to these posts is
> > that you are posting tcpdump-decoded data. Tcpdump still has no support
> > for NFSv4, and therefore completely garbles the output by trying to
> > interpret it as NFSv2/v3.
> > In general, if you are posting network traffic, please record it as
> > binary raw packet data (using the '-w' option on tcdump) so that we can
> > look at the full contents. Either include it as an attachment, or
> > provide us with details on how to download it from an http server.
> >
> > Other information that is needed in order to make sense of NFS bug
> > reports includes:
>
> Thank you Trond, I figured there was something missing but I didn't know
> where to start but here goes:
>
> >
> > - client OS (non-linux) or kernel version (linux)
> Client OS Linux 3.4.39, x86
>
> > - mount options on the client
> ~ # ypmatch jocke auto.home
> -fstype=nfs,soft devsrv:/mnt/home/jocke
>
> > - server OS (non-linux) or kernel version (linux)
> Server OS Linux 3.4.39, amd64
>
> > - type of exported filesystem on the server
> XFS
>
> > - contents of /etc/exports on the server
> more /etc/exports
> # /etc/exports: NFS file systems being exported. See exports(5).
> /mnt/home *(rw,async,root_squash,no_subtree_check)
> /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> /tftproot *(rw,async,root_squash,no_subtree_check)
> /mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
> /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
>
> /mnt/home is the one failing
>
> >
> > Please ensure that you always include those in your emails.
>
> nfs.pcap:
> http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
>
> nfs2.pcap:
> http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
>
> nfs3.pcap:
> http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
>
> nfs4.pcap:
> http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
>
> nfs3.pcap is the gvsd-metadata problem one can find using google, doesn't
> have to be a NFS problem
> The other 3 all come from surfing the www using firefox 17.0.3
The nfs2.pcap file and nfs4.pcap seem to show the server returning
NFS4ERR_OLD_STATEID, which usually means that the client has an
OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server has
updated the stateid, the client has not yet received the reply. The
problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
The nfs.pcap file is resending a load of LOCK requests that are
receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the recovery
engine to kick in and try to recover the OPEN.
So when you do 'ps -efwww', on any of these clients, do you see a
process with a name containing the server IP address (192.168.201.44)?
Also, is there anything special in the log when you do 'dmesg -s 90000'?
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@netapp.com
www.netapp.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-16 22:06 ` Myklebust, Trond
@ 2013-04-16 22:43 ` Joakim Tjernlund
2013-04-17 12:03 ` Joakim Tjernlund
` (2 subsequent siblings)
3 siblings, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-16 22:43 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/17
00:06:51:
>
> On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/16
> > 17:36:55:
> >
> > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > Date: 2013/04/16 17:37
> > > Subject: Re: NFS loop on 3.4.39
> > >
> > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > Here we go again, this time i happened while browsing the Boston
news
> > on
> > > > www.dn.se
> > > > Now gvfsd-metadata is turned off(not running at all) and I get:
> > > > 10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838:
reply
> > ok
> > > > 52 getattr ERROR: unk 10024
> > >
> > > Part of the reason why you are getting no response to these posts is
> > > that you are posting tcpdump-decoded data. Tcpdump still has no
support
> > > for NFSv4, and therefore completely garbles the output by trying to
> > > interpret it as NFSv2/v3.
> > > In general, if you are posting network traffic, please record it as
> > > binary raw packet data (using the '-w' option on tcdump) so that we
can
> > > look at the full contents. Either include it as an attachment, or
> > > provide us with details on how to download it from an http server.
> > >
> > > Other information that is needed in order to make sense of NFS bug
> > > reports includes:
> >
> > Thank you Trond, I figured there was something missing but I didn't
know
> > where to start but here goes:
> >
> > >
> > > - client OS (non-linux) or kernel version (linux)
> > Client OS Linux 3.4.39, x86
> >
> > > - mount options on the client
> > ~ # ypmatch jocke auto.home
> > -fstype=nfs,soft devsrv:/mnt/home/jocke
> >
> > > - server OS (non-linux) or kernel version (linux)
> > Server OS Linux 3.4.39, amd64
> >
> > > - type of exported filesystem on the server
> > XFS
> >
> > > - contents of /etc/exports on the server
> > more /etc/exports
> > # /etc/exports: NFS file systems being exported. See exports(5).
> > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > /tftproot *(rw,async,root_squash,no_subtree_check)
> > /mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
> > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> >
> > /mnt/home is the one failing
> >
> > >
> > > Please ensure that you always include those in your emails.
> >
> > nfs.pcap:
> > http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> >
> > nfs2.pcap:
> > http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> >
> > nfs3.pcap:
> > http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> >
> > nfs4.pcap:
> > http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> >
> > nfs3.pcap is the gvsd-metadata problem one can find using google,
doesn't
> > have to be a NFS problem
> > The other 3 all come from surfing the www using firefox 17.0.3
>
> The nfs2.pcap file and nfs4.pcap seem to show the server returning
> NFS4ERR_OLD_STATEID, which usually means that the client has an
> OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server has
> updated the stateid, the client has not yet received the reply. The
> problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
That figures, I only can only start tcpdump after the problem has occurred
as
it is not so easy to trigger.
>
> The nfs.pcap file is resending a load of LOCK requests that are
> receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the recovery
> engine to kick in and try to recover the OPEN.
>
> So when you do 'ps -efwww', on any of these clients, do you see a
> process with a name containing the server IP address (192.168.201.44)?
When I have the error you mean? Will have to do that the next time.
ATM I don't have any such processes, the only this I have save is:
ps uaxww| grep firefox
jocke 6330 0.2 0.0 0 0 tty1 Zl Apr15 3:04 [firefox]
<defunct>
jocke 10655 0.0 6.8 733648 279432 tty1 D 10:25 0:00 firefox
jocke 10691 0.0 6.8 733648 279432 tty1 D 10:26 0:00 firefox
Strangely there are 3 ff processes
>
> Also, is there anything special in the log when you do 'dmesg -s 90000'?
Don't have the problem ATM but I did save a dmesg form nfs4.pcap run:
[io 0x0089-0x008b]
pnp 00:05: [io 0x008f]
pnp 00:05: [io 0x00c0-0x00df]
pnp 00:05: Plug and Play ACPI device, IDs PNP0200 (active)
pnp 00:06: [io 0x0070-0x0071]
pnp 00:06: [irq 8]
pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active)
pnp 00:07: [io 0x0061]
pnp 00:07: Plug and Play ACPI device, IDs PNP0800 (active)
pnp 00:08: [io 0x0010-0x001f]
pnp 00:08: [io 0x0022-0x003f]
pnp 00:08: [io 0x0044-0x005f]
pnp 00:08: [io 0x0062-0x0063]
pnp 00:08: [io 0x0065-0x006f]
pnp 00:08: [io 0x0072-0x007f]
pnp 00:08: [io 0x0080]
pnp 00:08: [io 0x0084-0x0086]
pnp 00:08: [io 0x0088]
pnp 00:08: [io 0x008c-0x008e]
pnp 00:08: [io 0x0090-0x009f]
pnp 00:08: [io 0x00a2-0x00bf]
pnp 00:08: [io 0x00e0-0x00ef]
pnp 00:08: [io 0x04d0-0x04d1]
system 00:08: [io 0x04d0-0x04d1] has been reserved
system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:09: [io 0x00f0-0x00ff]
pnp 00:09: [irq 13]
pnp 00:09: Plug and Play ACPI device, IDs PNP0c04 (active)
pnp 00:0a: [io 0x03f8-0x03ff]
pnp 00:0a: [irq 4]
pnp 00:0a: [dma 0 disabled]
pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active)
pnp 00:0b: [mem 0xfed40000-0xfed44fff]
pnp 00:0b: Plug and Play ACPI device, IDs IFX0102 PNP0c31 (active)
pnp 00:0c: [io 0x0400-0x0453]
pnp 00:0c: [io 0x0458-0x047f]
pnp 00:0c: [io 0x1180-0x119f]
pnp 00:0c: [io 0x0500-0x057f]
pnp 00:0c: [mem 0xfed1c000-0xfed1ffff]
pnp 00:0c: [mem 0xfec00000-0xfecfffff]
pnp 00:0c: [mem 0xfed08000-0xfed08fff]
pnp 00:0c: [mem 0xff000000-0xffffffff]
system 00:0c: [io 0x0400-0x0453] has been reserved
system 00:0c: [io 0x0458-0x047f] has been reserved
system 00:0c: [io 0x1180-0x119f] has been reserved
system 00:0c: [io 0x0500-0x057f] has been reserved
system 00:0c: [mem 0xfed1c000-0xfed1ffff] has been reserved
system 00:0c: [mem 0xfec00000-0xfecfffff] could not be reserved
system 00:0c: [mem 0xfed08000-0xfed08fff] has been reserved
system 00:0c: [mem 0xff000000-0xffffffff] has been reserved
system 00:0c: Plug and Play ACPI device, IDs PNP0c01 (active)
pnp 00:0d: [io 0x0454-0x0457]
system 00:0d: [io 0x0454-0x0457] has been reserved
system 00:0d: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
pnp 00:0e: [mem 0xfed00000-0xfed003ff]
pnp 00:0e: Plug and Play ACPI device, IDs PNP0103 (active)
pnp: PnP ACPI: found 15 devices
ACPI: ACPI bus type pnp unregistered
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0: bridge window [io 0xe000-0xefff]
pci 0000:00:01.0: bridge window [mem 0xfe600000-0xfe6fffff]
pci 0000:00:01.0: bridge window [mem 0xd0000000-0xdfffffff 64bit pref]
pci 0000:00:1c.0: PCI bridge to [bus 02-02]
pci 0000:00:1c.4: PCI bridge to [bus 03-03]
pci 0000:00:1c.6: PCI bridge to [bus 04-04]
pci 0000:00:1c.7: PCI bridge to [bus 05-05]
pci 0000:00:1e.0: PCI bridge to [bus 06-06]
pci 0000:00:1e.0: bridge window [io 0xd000-0xdfff]
pci 0000:00:1e.0: bridge window [mem 0xfe500000-0xfe5fffff]
pci 0000:00:1e.0: setting latency timer to 64
pci_bus 0000:00: resource 4 [io 0x0000-0x03af]
pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7]
pci_bus 0000:00: resource 6 [io 0x03b0-0x03df]
pci_bus 0000:00: resource 7 [io 0x0d00-0xffff]
pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff]
pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff]
pci_bus 0000:00: resource 10 [mem 0xd0000000-0xffffffff]
pci_bus 0000:01: resource 0 [io 0xe000-0xefff]
pci_bus 0000:01: resource 1 [mem 0xfe600000-0xfe6fffff]
pci_bus 0000:01: resource 2 [mem 0xd0000000-0xdfffffff 64bit pref]
pci_bus 0000:06: resource 0 [io 0xd000-0xdfff]
pci_bus 0000:06: resource 1 [mem 0xfe500000-0xfe5fffff]
pci_bus 0000:06: resource 4 [io 0x0000-0x03af]
pci_bus 0000:06: resource 5 [io 0x03e0-0x0cf7]
pci_bus 0000:06: resource 6 [io 0x03b0-0x03df]
pci_bus 0000:06: resource 7 [io 0x0d00-0xffff]
pci_bus 0000:06: resource 8 [mem 0x000a0000-0x000bffff]
pci_bus 0000:06: resource 9 [mem 0x000c0000-0x000dffff]
pci_bus 0000:06: resource 10 [mem 0xd0000000-0xffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP: reno registered
UDP hash table entries: 512 (order: 2, 16384 bytes)
UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
pci 0000:01:00.0: Boot video device
PCI: CLS 64 bytes, default 64
audit: initializing netlink socket (disabled)
type=2000 audit(1366021073.520:1): initialized
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
NFS: Registering the id_resolver key type
FS-Cache: Netfs 'nfs' registered for caching
SGI XFS with ACLs, security attributes, realtime, large block/inode
numbers, no debug enabled
msgmni has been set to 1670
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler deadline registered (default)
pcieport 0000:00:01.0: irq 40 for MSI/MSI-X
pcieport 0000:00:1c.0: irq 41 for MSI/MSI-X
pcieport 0000:00:1c.4: irq 42 for MSI/MSI-X
pcieport 0000:00:1c.6: irq 43 for MSI/MSI-X
pcieport 0000:00:1c.7: irq 44 for MSI/MSI-X
pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
pci 0000:01:00.1: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded
pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded
pcieport 0000:00:1c.4: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:1c.4:pcie01: service driver pcie_pme loaded
pcieport 0000:00:1c.6: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:1c.6:pcie01: service driver pcie_pme loaded
pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:1c.7:pcie01: service driver pcie_pme loaded
intel_idle: MWAIT substates: 0x1120
intel_idle: v0.4 model 0x2A
intel_idle: lapic_timer_reliable_states 0xffffffff
input: Power Button as
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
ACPI: Power Button [PWRB]
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
ACPI: Power Button [PWRF]
ACPI: Requesting acpi_cpufreq
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
0000:00:16.3: ttyS1 at I/O 0xf0a0 (irq = 17) is a 16550A
0000:06:01.0: ttyS2 at I/O 0xd020 (irq = 21) is a 16550A
Linux agpgart interface v0.103
[drm] Initialized drm 1.1.0 20060810
Refined TSC clocksource calibration: 3392.286 MHz.
Switching to clocksource tsc
floppy0: no floppy controllers found
brd: module loaded
loop: module loaded
Compaq SMART2 Driver (v 2.6.0)
HP CISS Driver (v 3.6.26)
Adaptec aacraid driver 1.2-0[28900]-ms
st: Version 20101219, fixed bufsize 32768, s/g segs 256
ahci 0000:00:1f.2: version 3.0
ahci 0000:00:1f.2: irq 45 for MSI/MSI-X
ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x15 impl SATA
mode
ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems sxs
apst
ahci 0000:00:1f.2: setting latency timer to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 abar m2048@0xfe725000 port 0xfe725100 irq 45
ata2: DUMMY
ata3: SATA max UDMA/133 abar m2048@0xfe725000 port 0xfe725200 irq 45
ata4: DUMMY
ata5: SATA max UDMA/133 abar m2048@0xfe725000 port 0xfe725300 irq 45
ata6: DUMMY
e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
e1000: Copyright (c) 1999-2006 Intel Corporation.
e1000e: Intel(R) PRO/1000 Network Driver - 1.9.5-k
e1000e: Copyright(c) 1999 - 2012 Intel Corporation.
e1000e 0000:00:19.0: setting latency timer to 64
e1000e 0000:00:19.0: (unregistered net_device): Interrupt Throttling Rate
(ints/sec) set to dynamic conservative mode
e1000e 0000:00:19.0: irq 46 for MSI/MSI-X
e1000e 0000:00:19.0: eth0: (PCI Express:2.5GT/s:Width x1)
08:2e:5f:1b:3d:98
e1000e 0000:00:19.0: eth0: Intel(R) PRO/1000 Network Connection
e1000e 0000:00:19.0: eth0: MAC: 10, PHY: 11, PBA No: 0100FF-0FF
igb: Intel(R) Gigabit Ethernet Network Driver - version 3.2.10-k
igb: Copyright (c) 2007-2012 Intel Corporation.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:1a.0: setting latency timer to 64
ehci_hcd 0000:00:1a.0: EHCI Host Controller
ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.0: debug port 2
ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1a.0: irq 16, io mem 0xfe727000
ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
ehci_hcd 0000:00:1d.0: setting latency timer to 64
ehci_hcd 0000:00:1d.0: EHCI Host Controller
ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.0: debug port 2
ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1d.0: irq 23, io mem 0xfe726000
ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
uhci_hcd: USB Universal Host Controller Interface driver
i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq
1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mousedev: PS/2 mouse device common for all mice
input: PC Speaker as /devices/platform/pcspkr/input/input2
device-mapper: ioctl: 4.22.0-ioctl (2011-10-19) initialised:
dm-devel@redhat.com
EISA: Probing bus 0 at eisa.0
EISA: Cannot allocate resource for mainboard
Cannot allocate resource for EISA slot 1
Cannot allocate resource for EISA slot 2
Cannot allocate resource for EISA slot 3
Cannot allocate resource for EISA slot 4
Cannot allocate resource for EISA slot 5
Cannot allocate resource for EISA slot 6
Cannot allocate resource for EISA slot 7
Cannot allocate resource for EISA slot 8
EISA: Detected 0 cards.
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP: cubic registered
NET: Registered protocol family 10
NET: Registered protocol family 17
Bridge firewalling registered
8021q: 802.1Q VLAN Support v1.8
TIPC: Activated (version 2.0.0)
NET: Registered protocol family 30
TIPC: Started in single node mode
Registering the dns_resolver key type
openvswitch: Open vSwitch switching datapath
Using IPI No-Shortcut mode
registered taskstats version 1
ALSA device list:
No soundcards found.
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata5: SATA link down (SStatus 0 SControl 300)
ata3.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata3.00: ATAPI: hp DVD RW AD-7250H5, 1HNK, max UDMA/100
ata3.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata3.00: configured for UDMA/100
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata1.00: ATA-8: ST3500413AS, HP64, max UDMA/100
ata1.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32)
ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
ata1.00: configured for UDMA/100
scsi 0:0:0:0: Direct-Access ATA ST3500413AS HP64 PQ: 0 ANSI:
5
sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't
support DPO or FUA
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 2:0:0:0: CD-ROM hp DVD RW AD-7250H5 1HNK PQ: 0 ANSI:
5
sr0: scsi3-mmc drive: 40x/40x writer dvd-ram cd/rw xa/form2 cdda tray
cdrom: Uniform CD-ROM driver Revision: 3.20
sr 2:0:0:0: Attached scsi CD-ROM sr0
sr 2:0:0:0: Attached scsi generic sg1 type 5
sda: sda1 sda2 sda3 sda4
sd 0:0:0:0: [sda] Attached SCSI disk
usb 1-1: new high-speed USB device number 2 using ehci_hcd
hub 1-1:1.0: USB hub found
hub 1-1:1.0: 6 ports detected
usb 2-1: new high-speed USB device number 2 using ehci_hcd
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 8 ports detected
input: ImPS/2 Generic Wheel Mouse as
/devices/platform/i8042/serio1/input/input3
usb 1-1.5: new low-speed USB device number 3 using ehci_hcd
XFS (sda3): Mounting Filesystem
input: CHICONY HP Basic USB Keyboard as
/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.5/1-1.5:1.0/input/input4
generic-usb 0003:03F0:0024.0001: input: USB HID v1.11 Keyboard [CHICONY HP
Basic USB Keyboard] on usb-0000:00:1a.0-1.5/input0
XFS (sda3): Starting recovery (logdev: internal)
XFS (sda3): Ending recovery (logdev: internal)
VFS: Mounted root (xfs filesystem) readonly on device 8:3.
Freeing unused kernel memory: 424k freed
systemd-udevd[1205]: starting version 197
microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x1b
snd_hda_intel 0000:00:1b.0: irq 47 for MSI/MSI-X
microcode: CPU1 sig=0x206a7, pf=0x2, revision=0x1b
microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x1b
microcode: CPU3 sig=0x206a7, pf=0x2, revision=0x1b
microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>,
Peter Oruba
snd_hda_intel 0000:01:00.1: irq 48 for MSI/MSI-X
[drm] radeon defaulting to kernel modesetting.
[drm] radeon kernel modesetting enabled.
[drm] initializing kernel modesetting (TURKS 0x1002:0x6759 0x103C:0x3130).
[drm] register mmio base: 0xFE620000
[drm] register mmio size: 131072
ATOM BIOS: Bobafet
radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
(1024M used)
radeon 0000:01:00.0: GTT: 512M 0x0000000040000000 - 0x000000005FFFFFFF
[drm] Detected VRAM RAM=1024M, BAR=256M
[drm] RAM width 128bits DDR
[TTM] Zone kernel: Available graphics memory: 427752 kiB
[TTM] Zone highmem: Available graphics memory: 2046914 kiB
[TTM] Initializing pool allocator
[drm] radeon: 1024M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
[drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[drm] Driver supports precise vblank timestamp query.
radeon 0000:01:00.0: irq 49 for MSI/MSI-X
radeon 0000:01:00.0: radeon: using MSI.
[drm] radeon: irq initialized.
[drm] GART: num cpu pages 131072, num gpu pages 131072
[drm] radeon: ib pool ready.
[drm] Loading TURKS Microcode
[drm] PCIE GART of 512M enabled (table at 0x0000000000040000).
radeon 0000:01:00.0: WB enabled
[drm] fence driver on ring 0 use gpu addr 0x40000c00 and cpu addr
0xffceec00
[drm] ring test on 0 succeeded in 3 usecs
[drm] ib test on ring 0 succeeded in 0 usecs
[drm] Radeon Display Connectors
[drm] Connector 0:
[drm] DisplayPort
[drm] HPD4
[drm] DDC: 0x6450 0x6450 0x6454 0x6454 0x6458 0x6458 0x645c 0x645c
[drm] Encoders:
[drm] DFP1: INTERNAL_UNIPHY2
[drm] Connector 1:
[drm] DisplayPort
[drm] HPD5
[drm] DDC: 0x6470 0x6470 0x6474 0x6474 0x6478 0x6478 0x647c 0x647c
[drm] Encoders:
[drm] DFP2: INTERNAL_UNIPHY2
[drm] Connector 2:
[drm] DVI-I
[drm] HPD1
[drm] DDC: 0x6460 0x6460 0x6464 0x6464 0x6468 0x6468 0x646c 0x646c
[drm] Encoders:
[drm] DFP3: INTERNAL_UNIPHY
[drm] CRT1: INTERNAL_KLDSCP_DAC1
[drm] Internal thermal controller with fan control
[drm] radeon: power management initialized
[drm] fb mappable at 0xD0142000
[drm] vram apper at 0xD0000000
[drm] size 9216000
[drm] fb depth is 24
[drm] pitch is 7680
fbcon: radeondrmfb (fb0) is primary device
Console: switching to colour frame buffer device 240x75
fb0: radeondrmfb frame buffer device
drm: registered panic notifier
[drm] Initialized radeon 2.16.0 20080528 for 0000:01:00.0 on minor 0
Adding 4194300k swap on /dev/sda2. Priority:-1 extents:1 across:4194300k
EXT4-fs (sda1): barriers disabled
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: nobarrier
EXT4-fs (dm-2): VFS: Can't find ext4 filesystem
FAT-fs (dm-2): Unrecognized mount option "nobarrier" or missing value
XFS (dm-2): Mounting Filesystem
XFS (dm-2): Starting recovery (logdev: internal)
XFS (dm-2): Ending recovery (logdev: internal)
e1000e 0000:00:19.0: irq 46 for MSI/MSI-X
e1000e 0000:00:19.0: irq 46 for MSI/MSI-X
ADDRCONF(NETDEV_UP): eth0: link is not ready
e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
eth0: no IPv6 routers present
ata1.00: configured for UDMA/100
ata1: EH complete
EXT4-fs (sda1): re-mounted. Opts: nobarrier,commit=0
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000000
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
INFO: task Gecko_IOThread:10655 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10655 6336 0x00000004
f44a3400 00000086 ece3dea0 00000000 00000000 c10cb15b ef6fe030 c17e4280
f30c17c0 ece3de60 c17e4280 c17e4280 ef6fe030 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434638 f4434638
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task Gecko_IOThread:10691 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10691 6336 0x00000000
f44a3400 00000086 d0247ea0 00000000 00000000 c10cb15b ef317400 c17e4280
f31590c0 d0247e60 c17e4280 c17e4280 ef317400 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434ae8 f4434ae8
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000004
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
INFO: task Gecko_IOThread:10655 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10655 6355 0x00000004
f44a3400 00000086 ece3dea0 00000000 00000000 c10cb15b ef6fe030 c17e4280
f30c17c0 ece3de60 c17e4280 c17e4280 ef6fe030 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434638 f4434638
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task Gecko_IOThread:10691 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10691 6355 0x00000000
f44a3400 00000086 d0247ea0 00000000 00000000 c10cb15b ef317400 c17e4280
f31590c0 d0247e60 c17e4280 c17e4280 ef317400 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434ae8 f4434ae8
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000004
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
INFO: task Gecko_IOThread:10655 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10655 6355 0x00000004
f44a3400 00000086 ece3dea0 00000000 00000000 c10cb15b ef6fe030 c17e4280
f30c17c0 ece3de60 c17e4280 c17e4280 ef6fe030 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434638 f4434638
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task Gecko_IOThread:10691 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10691 6355 0x00000000
f44a3400 00000086 d0247ea0 00000000 00000000 c10cb15b ef317400 c17e4280
f31590c0 d0247e60 c17e4280 c17e4280 ef317400 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434ae8 f4434ae8
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
nfsd: last server has exited, flushing export cache
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000004
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-16 22:06 ` Myklebust, Trond
2013-04-16 22:43 ` Joakim Tjernlund
@ 2013-04-17 12:03 ` Joakim Tjernlund
2013-04-18 12:34 ` Joakim Tjernlund
[not found] ` <OF97D68E61.A3C35530-ONC1257B51.0043E567-C1257B51.00450958@LocalDomain>
3 siblings, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-17 12:03 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/17
00:06:51:
>
> On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/16
> > 17:36:55:
> >
> > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > Date: 2013/04/16 17:37
> > > Subject: Re: NFS loop on 3.4.39
> > >
> > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > Here we go again, this time i happened while browsing the Boston
news
> > on
> > > > www.dn.se
> > > > Now gvfsd-metadata is turned off(not running at all) and I get:
> > > > 10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838:
reply
> > ok
> > > > 52 getattr ERROR: unk 10024
> > >
> > > Part of the reason why you are getting no response to these posts is
> > > that you are posting tcpdump-decoded data. Tcpdump still has no
support
> > > for NFSv4, and therefore completely garbles the output by trying to
> > > interpret it as NFSv2/v3.
> > > In general, if you are posting network traffic, please record it as
> > > binary raw packet data (using the '-w' option on tcdump) so that we
can
> > > look at the full contents. Either include it as an attachment, or
> > > provide us with details on how to download it from an http server.
> > >
> > > Other information that is needed in order to make sense of NFS bug
> > > reports includes:
> >
> > Thank you Trond, I figured there was something missing but I didn't
know
> > where to start but here goes:
> >
> > >
> > > - client OS (non-linux) or kernel version (linux)
> > Client OS Linux 3.4.39, x86
> >
> > > - mount options on the client
> > ~ # ypmatch jocke auto.home
> > -fstype=nfs,soft devsrv:/mnt/home/jocke
> >
> > > - server OS (non-linux) or kernel version (linux)
> > Server OS Linux 3.4.39, amd64
> >
> > > - type of exported filesystem on the server
> > XFS
> >
> > > - contents of /etc/exports on the server
> > more /etc/exports
> > # /etc/exports: NFS file systems being exported. See exports(5).
> > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > /tftproot *(rw,async,root_squash,no_subtree_check)
> > /mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
> > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> >
> > /mnt/home is the one failing
> >
> > >
> > > Please ensure that you always include those in your emails.
> >
> > nfs.pcap:
> > http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> >
> > nfs2.pcap:
> > http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> >
> > nfs3.pcap:
> > http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> >
> > nfs4.pcap:
> > http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> >
> > nfs3.pcap is the gvsd-metadata problem one can find using google,
doesn't
> > have to be a NFS problem
> > The other 3 all come from surfing the www using firefox 17.0.3
>
> The nfs2.pcap file and nfs4.pcap seem to show the server returning
> NFS4ERR_OLD_STATEID, which usually means that the client has an
> OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server has
> updated the stateid, the client has not yet received the reply. The
> problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
>
> The nfs.pcap file is resending a load of LOCK requests that are
> receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the recovery
> engine to kick in and try to recover the OPEN.
>
> So when you do 'ps -efwww', on any of these clients, do you see a
> process with a name containing the server IP address (192.168.201.44)?
>
> Also, is there anything special in the log when you do 'dmesg -s 90000'?
ATM, I cannot reproduce the NFS problem. Don't know what to about it.
If it happens again I can start tcpdump but I guess you will
miss the initial NFS traffic again in that case.
Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-16 22:06 ` Myklebust, Trond
2013-04-16 22:43 ` Joakim Tjernlund
2013-04-17 12:03 ` Joakim Tjernlund
@ 2013-04-18 12:34 ` Joakim Tjernlund
[not found] ` <OF97D68E61.A3C35530-ONC1257B51.0043E567-C1257B51.00450958@LocalDomain>
3 siblings, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-18 12:34 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/17
00:06:51:
>
> On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/16
> > 17:36:55:
> >
> > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > Date: 2013/04/16 17:37
> > > Subject: Re: NFS loop on 3.4.39
> > >
> > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > Here we go again, this time i happened while browsing the Boston
news
> > on
> > > > www.dn.se
> > > > Now gvfsd-metadata is turned off(not running at all) and I get:
> > > > 10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838:
reply
> > ok
> > > > 52 getattr ERROR: unk 10024
> > >
> > > Part of the reason why you are getting no response to these posts is
> > > that you are posting tcpdump-decoded data. Tcpdump still has no
support
> > > for NFSv4, and therefore completely garbles the output by trying to
> > > interpret it as NFSv2/v3.
> > > In general, if you are posting network traffic, please record it as
> > > binary raw packet data (using the '-w' option on tcdump) so that we
can
> > > look at the full contents. Either include it as an attachment, or
> > > provide us with details on how to download it from an http server.
> > >
> > > Other information that is needed in order to make sense of NFS bug
> > > reports includes:
> >
> > Thank you Trond, I figured there was something missing but I didn't
know
> > where to start but here goes:
> >
> > >
> > > - client OS (non-linux) or kernel version (linux)
> > Client OS Linux 3.4.39, x86
> >
> > > - mount options on the client
> > ~ # ypmatch jocke auto.home
> > -fstype=nfs,soft devsrv:/mnt/home/jocke
> >
> > > - server OS (non-linux) or kernel version (linux)
> > Server OS Linux 3.4.39, amd64
> >
> > > - type of exported filesystem on the server
> > XFS
> >
> > > - contents of /etc/exports on the server
> > more /etc/exports
> > # /etc/exports: NFS file systems being exported. See exports(5).
> > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > /tftproot *(rw,async,root_squash,no_subtree_check)
> > /mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
> > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> >
> > /mnt/home is the one failing
> >
> > >
> > > Please ensure that you always include those in your emails.
> >
> > nfs.pcap:
> > http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> >
> > nfs2.pcap:
> > http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> >
> > nfs3.pcap:
> > http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> >
> > nfs4.pcap:
> > http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> >
> > nfs3.pcap is the gvsd-metadata problem one can find using google,
doesn't
> > have to be a NFS problem
> > The other 3 all come from surfing the www using firefox 17.0.3
>
> The nfs2.pcap file and nfs4.pcap seem to show the server returning
> NFS4ERR_OLD_STATEID, which usually means that the client has an
> OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server has
> updated the stateid, the client has not yet received the reply. The
> problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
>
> The nfs.pcap file is resending a load of LOCK requests that are
> receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the recovery
> engine to kick in and try to recover the OPEN.
>
> So when you do 'ps -efwww', on any of these clients, do you see a
> process with a name containing the server IP address (192.168.201.44)?
>
> Also, is there anything special in the log when you do 'dmesg -s 90000'?
Of course this happened again while I wasn't looking so I don't know what
caused it, probably firefox though.
There is nothing in dmesg and ps -efwww has no hit on IP
address 192.168.201.44, the closest I can get is:
ps -efwww | grep nfs
root 568 2 0 Apr16 ? 00:00:00 [nfsiod]
root 2440 2 0 Apr16 ? 00:00:00 [nfsd4]
root 2441 2 0 Apr16 ? 00:00:00 [nfsd4_callbacks]
root 2442 2 0 Apr16 ? 00:00:00 [nfsd]
root 2443 2 0 Apr16 ? 00:00:00 [nfsd]
root 2444 2 0 Apr16 ? 00:00:00 [nfsd]
root 2445 2 0 Apr16 ? 00:00:00 [nfsd]
root 2446 2 0 Apr16 ? 00:00:00 [nfsd]
root 2447 2 0 Apr16 ? 00:00:00 [nfsd]
root 2448 2 0 Apr16 ? 00:00:00 [nfsd]
root 2449 2 0 Apr16 ? 00:00:00 [nfsd]
root 2667 2 0 Apr16 ? 00:00:00 [nfsv4.0-svc]
jocke 27048 26888 0 14:28 pts/3 00:00:00 grep --colour=auto nfs
Got a new pcap file also:
http://ftp-us.transmode.se/get/?id=6f935e1d7e105d01e9a5b907c6493521
nfs5.pcap
The load is not that noticeable so I can stay in this mode a while, until
I go home today.
Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
[not found] ` <OF97D68E61.A3C35530-ONC1257B51.0043E567-C1257B51.00450958@LocalDomain>
@ 2013-04-19 10:54 ` Joakim Tjernlund
[not found] ` <OFF8AA22C1.D96F08FD-ONC1257B52.003B6FD9-C1257B52.003BEEF9@LocalDomain>
1 sibling, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-19 10:54 UTC (permalink / raw)
Cc: Myklebust, Trond, linux-nfs
Joakim Tjernlund/Transmode wrote on 2013/04/18 14:34:03:
>
> "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/17
00:06:51:
> >
> > On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/16
> > > 17:36:55:
> > >
> > > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > > Date: 2013/04/16 17:37
> > > > Subject: Re: NFS loop on 3.4.39
> > > >
> > > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > > Here we go again, this time i happened while browsing the Boston
news
> > > on
> > > > > www.dn.se
> > > > > Now gvfsd-metadata is turned off(not running at all) and I get:
> > > > > 10:28:44.616146 IP 192.168.201.44.nfs > 172.20.4.10.3671768838:
reply
> > > ok
> > > > > 52 getattr ERROR: unk 10024
> > > >
> > > > Part of the reason why you are getting no response to these posts
is
> > > > that you are posting tcpdump-decoded data. Tcpdump still has no
support
> > > > for NFSv4, and therefore completely garbles the output by trying
to
> > > > interpret it as NFSv2/v3.
> > > > In general, if you are posting network traffic, please record it
as
> > > > binary raw packet data (using the '-w' option on tcdump) so that
we can
> > > > look at the full contents. Either include it as an attachment, or
> > > > provide us with details on how to download it from an http server.
> > > >
> > > > Other information that is needed in order to make sense of NFS bug
> > > > reports includes:
> > >
> > > Thank you Trond, I figured there was something missing but I didn't
know
> > > where to start but here goes:
> > >
> > > >
> > > > - client OS (non-linux) or kernel version (linux)
> > > Client OS Linux 3.4.39, x86
> > >
> > > > - mount options on the client
> > > ~ # ypmatch jocke auto.home
> > > -fstype=nfs,soft devsrv:/mnt/home/jocke
> > >
> > > > - server OS (non-linux) or kernel version (linux)
> > > Server OS Linux 3.4.39, amd64
> > >
> > > > - type of exported filesystem on the server
> > > XFS
> > >
> > > > - contents of /etc/exports on the server
> > > more /etc/exports
> > > # /etc/exports: NFS file systems being exported. See exports(5).
> > > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > > /tftproot *(rw,async,root_squash,no_subtree_check)
> > > /mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
> > > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> > >
> > > /mnt/home is the one failing
> > >
> > > >
> > > > Please ensure that you always include those in your emails.
> > >
> > > nfs.pcap:
> > > http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> > >
> > > nfs2.pcap:
> > > http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> > >
> > > nfs3.pcap:
> > > http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> > >
> > > nfs4.pcap:
> > > http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> > >
> > > nfs3.pcap is the gvsd-metadata problem one can find using google,
doesn't
> > > have to be a NFS problem
> > > The other 3 all come from surfing the www using firefox 17.0.3
> >
> > The nfs2.pcap file and nfs4.pcap seem to show the server returning
> > NFS4ERR_OLD_STATEID, which usually means that the client has an
> > OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server has
> > updated the stateid, the client has not yet received the reply. The
> > problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
> >
> > The nfs.pcap file is resending a load of LOCK requests that are
> > receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the
recovery
> > engine to kick in and try to recover the OPEN.
> >
> > So when you do 'ps -efwww', on any of these clients, do you see a
> > process with a name containing the server IP address (192.168.201.44)?
> >
> > Also, is there anything special in the log when you do 'dmesg -s
90000'?
> Of course this happened again while I wasn't looking so I don't know
what
> caused it, probably firefox though.
>
> There is nothing in dmesg and ps -efwww has no hit on IP
> address 192.168.201.44, the closest I can get is:
> ps -efwww | grep nfs
> root 568 2 0 Apr16 ? 00:00:00 [nfsiod]
> root 2440 2 0 Apr16 ? 00:00:00 [nfsd4]
> root 2441 2 0 Apr16 ? 00:00:00 [nfsd4_callbacks]
> root 2442 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2443 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2444 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2445 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2446 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2447 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2448 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2449 2 0 Apr16 ? 00:00:00 [nfsd]
> root 2667 2 0 Apr16 ? 00:00:00 [nfsv4.0-svc]
> jocke 27048 26888 0 14:28 pts/3 00:00:00 grep --colour=auto nfs
>
> Got a new pcap file also:
> http://ftp-us.transmode.se/get/?id=6f935e1d7e105d01e9a5b907c6493521
nfs5.pcap
>
> The load is not that noticeable so I can stay in this mode a while,
until I go
> home today.
So left it overnight and this morning my NFS client had completely looked
up,
had to press the power button. This has happened twice now.
One more piece of info, we think this problem started when NFS server
was upgraded from 3.4.28 to 3.4.39
I have no idea how to move forward now. Trond, are you also stuck?
Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
[not found] ` <OFF8AA22C1.D96F08FD-ONC1257B52.003B6FD9-C1257B52.003BEEF9@LocalDomain>
@ 2013-04-23 13:38 ` Joakim Tjernlund
2013-04-23 13:52 ` Myklebust, Trond
0 siblings, 1 reply; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-23 13:38 UTC (permalink / raw)
Cc: Myklebust, Trond, linux-nfs
So, it happened again. Just when hitting search on bugs.gentoo.org in
firefox 17.0.3
This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over and over
again and FF was hung. Not posting the logs as it does not appear to
do any good. Nothing in dmesg either.
Noticed this patch on the NFS list:
http://marc.info/?l=linux-nfs&m=136643651710066&w=2
I wonder if that could be a potential cure and if so, could it be
backported to 3.4?
Jocke
Joakim Tjernlund/Transmode wrote on 2013/04/19 12:54:38:
>
> Joakim Tjernlund/Transmode wrote on 2013/04/18 14:34:03:
> >
> > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/17
00:06:51:
> > >
> > > On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
2013/04/16
> > > > 17:36:55:
> > > >
> > > > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > > > Date: 2013/04/16 17:37
> > > > > Subject: Re: NFS loop on 3.4.39
> > > > >
> > > > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > > > Here we go again, this time i happened while browsing the
Boston news
> > > > on
> > > > > > www.dn.se
> > > > > > Now gvfsd-metadata is turned off(not running at all) and I
get:
> > > > > > 10:28:44.616146 IP 192.168.201.44.nfs >
172.20.4.10.3671768838: reply
> > > > ok
> > > > > > 52 getattr ERROR: unk 10024
> > > > >
> > > > > Part of the reason why you are getting no response to these
posts is
> > > > > that you are posting tcpdump-decoded data. Tcpdump still has no
support
> > > > > for NFSv4, and therefore completely garbles the output by trying
to
> > > > > interpret it as NFSv2/v3.
> > > > > In general, if you are posting network traffic, please record it
as
> > > > > binary raw packet data (using the '-w' option on tcdump) so that
we can
> > > > > look at the full contents. Either include it as an attachment,
or
> > > > > provide us with details on how to download it from an http
server.
> > > > >
> > > > > Other information that is needed in order to make sense of NFS
bug
> > > > > reports includes:
> > > >
> > > > Thank you Trond, I figured there was something missing but I
didn't know
> > > > where to start but here goes:
> > > >
> > > > >
> > > > > - client OS (non-linux) or kernel version (linux)
> > > > Client OS Linux 3.4.39, x86
> > > >
> > > > > - mount options on the client
> > > > ~ # ypmatch jocke auto.home
> > > > -fstype=nfs,soft devsrv:/mnt/home/jocke
> > > >
> > > > > - server OS (non-linux) or kernel version (linux)
> > > > Server OS Linux 3.4.39, amd64
> > > >
> > > > > - type of exported filesystem on the server
> > > > XFS
> > > >
> > > > > - contents of /etc/exports on the server
> > > > more /etc/exports
> > > > # /etc/exports: NFS file systems being exported. See exports(5).
> > > > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > > > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > > > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > > > /tftproot *(rw,async,root_squash,no_subtree_check)
> > > > /mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
> > > > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> > > >
> > > > /mnt/home is the one failing
> > > >
> > > > >
> > > > > Please ensure that you always include those in your emails.
> > > >
> > > > nfs.pcap:
> > > >
http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> > > >
> > > > nfs2.pcap:
> > > >
http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> > > >
> > > > nfs3.pcap:
> > > >
http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> > > >
> > > > nfs4.pcap:
> > > >
http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> > > >
> > > > nfs3.pcap is the gvsd-metadata problem one can find using google,
doesn't
> > > > have to be a NFS problem
> > > > The other 3 all come from surfing the www using firefox 17.0.3
> > >
> > > The nfs2.pcap file and nfs4.pcap seem to show the server returning
> > > NFS4ERR_OLD_STATEID, which usually means that the client has an
> > > OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server has
> > > updated the stateid, the client has not yet received the reply. The
> > > problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
> > >
> > > The nfs.pcap file is resending a load of LOCK requests that are
> > > receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the
recovery
> > > engine to kick in and try to recover the OPEN.
> > >
> > > So when you do 'ps -efwww', on any of these clients, do you see a
> > > process with a name containing the server IP address
(192.168.201.44)?
> > >
> > > Also, is there anything special in the log when you do 'dmesg -s
90000'?
> > Of course this happened again while I wasn't looking so I don't know
what
> > caused it, probably firefox though.
> >
> > There is nothing in dmesg and ps -efwww has no hit on IP
> > address 192.168.201.44, the closest I can get is:
> > ps -efwww | grep nfs
> > root 568 2 0 Apr16 ? 00:00:00 [nfsiod]
> > root 2440 2 0 Apr16 ? 00:00:00 [nfsd4]
> > root 2441 2 0 Apr16 ? 00:00:00 [nfsd4_callbacks]
> > root 2442 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2443 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2444 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2445 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2446 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2447 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2448 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2449 2 0 Apr16 ? 00:00:00 [nfsd]
> > root 2667 2 0 Apr16 ? 00:00:00 [nfsv4.0-svc]
> > jocke 27048 26888 0 14:28 pts/3 00:00:00 grep --colour=auto nfs
> >
> > Got a new pcap file also:
> > http://ftp-us.transmode.se/get/?id=6f935e1d7e105d01e9a5b907c6493521
nfs5.pcap
> >
> > The load is not that noticeable so I can stay in this mode a while,
until I go
> > home today.
>
> So left it overnight and this morning my NFS client had completely
looked up,
> had to press the power button. This has happened twice now.
>
> One more piece of info, we think this problem started when NFS server
> was upgraded from 3.4.28 to 3.4.39
>
> I have no idea how to move forward now. Trond, are you also stuck?
>
> Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-23 13:38 ` Joakim Tjernlund
@ 2013-04-23 13:52 ` Myklebust, Trond
2013-04-23 14:14 ` Joakim Tjernlund
[not found] ` <OF940BEB3A.13ADB6E5-ONC1257B56.004D412D-C1257B56.004E3C54@tran <1366726687.35524.6.camel@leira.trondhjem.org>
0 siblings, 2 replies; 21+ messages in thread
From: Myklebust, Trond @ 2013-04-23 13:52 UTC (permalink / raw)
To: Joakim Tjernlund; +Cc: linux-nfs
On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> So, it happened again. Just when hitting search on bugs.gentoo.org in
> firefox 17.0.3
>
> This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over and over
> again and FF was hung. Not posting the logs as it does not appear to
> do any good. Nothing in dmesg either.
>
> Noticed this patch on the NFS list:
> http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> I wonder if that could be a potential cure and if so, could it be
> backported to 3.4?
It is in the testing branch on
http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
if you want to try it out. I'm not planning on backporting anything that
hasn't been labelled with a Cc: stable in that branch.
Cheers
Trond
> Jocke
>
> Joakim Tjernlund/Transmode wrote on 2013/04/19 12:54:38:
> >
> > Joakim Tjernlund/Transmode wrote on 2013/04/18 14:34:03:
> > >
> > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/17
> 00:06:51:
> > > >
> > > > On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
> 2013/04/16
> > > > > 17:36:55:
> > > > >
> > > > > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > > > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > > > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > > > > Date: 2013/04/16 17:37
> > > > > > Subject: Re: NFS loop on 3.4.39
> > > > > >
> > > > > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > > > > Here we go again, this time i happened while browsing the
> Boston news
> > > > > on
> > > > > > > www.dn.se
> > > > > > > Now gvfsd-metadata is turned off(not running at all) and I
> get:
> > > > > > > 10:28:44.616146 IP 192.168.201.44.nfs >
> 172.20.4.10.3671768838: reply
> > > > > ok
> > > > > > > 52 getattr ERROR: unk 10024
> > > > > >
> > > > > > Part of the reason why you are getting no response to these
> posts is
> > > > > > that you are posting tcpdump-decoded data. Tcpdump still has no
> support
> > > > > > for NFSv4, and therefore completely garbles the output by trying
> to
> > > > > > interpret it as NFSv2/v3.
> > > > > > In general, if you are posting network traffic, please record it
> as
> > > > > > binary raw packet data (using the '-w' option on tcdump) so that
> we can
> > > > > > look at the full contents. Either include it as an attachment,
> or
> > > > > > provide us with details on how to download it from an http
> server.
> > > > > >
> > > > > > Other information that is needed in order to make sense of NFS
> bug
> > > > > > reports includes:
> > > > >
> > > > > Thank you Trond, I figured there was something missing but I
> didn't know
> > > > > where to start but here goes:
> > > > >
> > > > > >
> > > > > > - client OS (non-linux) or kernel version (linux)
> > > > > Client OS Linux 3.4.39, x86
> > > > >
> > > > > > - mount options on the client
> > > > > ~ # ypmatch jocke auto.home
> > > > > -fstype=nfs,soft devsrv:/mnt/home/jocke
> > > > >
> > > > > > - server OS (non-linux) or kernel version (linux)
> > > > > Server OS Linux 3.4.39, amd64
> > > > >
> > > > > > - type of exported filesystem on the server
> > > > > XFS
> > > > >
> > > > > > - contents of /etc/exports on the server
> > > > > more /etc/exports
> > > > > # /etc/exports: NFS file systems being exported. See exports(5).
> > > > > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > > > > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > > > > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > > > > /tftproot *(rw,async,root_squash,no_subtree_check)
> > > > > /mnt/images *(rw,async,no_root_squash,no_subtree_check,insecure)
> > > > > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> > > > >
> > > > > /mnt/home is the one failing
> > > > >
> > > > > >
> > > > > > Please ensure that you always include those in your emails.
> > > > >
> > > > > nfs.pcap:
> > > > >
> http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> > > > >
> > > > > nfs2.pcap:
> > > > >
> http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> > > > >
> > > > > nfs3.pcap:
> > > > >
> http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> > > > >
> > > > > nfs4.pcap:
> > > > >
> http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> > > > >
> > > > > nfs3.pcap is the gvsd-metadata problem one can find using google,
> doesn't
> > > > > have to be a NFS problem
> > > > > The other 3 all come from surfing the www using firefox 17.0.3
> > > >
> > > > The nfs2.pcap file and nfs4.pcap seem to show the server returning
> > > > NFS4ERR_OLD_STATEID, which usually means that the client has an
> > > > OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server has
> > > > updated the stateid, the client has not yet received the reply. The
> > > > problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
> > > >
> > > > The nfs.pcap file is resending a load of LOCK requests that are
> > > > receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the
> recovery
> > > > engine to kick in and try to recover the OPEN.
> > > >
> > > > So when you do 'ps -efwww', on any of these clients, do you see a
> > > > process with a name containing the server IP address
> (192.168.201.44)?
> > > >
> > > > Also, is there anything special in the log when you do 'dmesg -s
> 90000'?
>
> > > Of course this happened again while I wasn't looking so I don't know
> what
> > > caused it, probably firefox though.
> > >
> > > There is nothing in dmesg and ps -efwww has no hit on IP
> > > address 192.168.201.44, the closest I can get is:
> > > ps -efwww | grep nfs
> > > root 568 2 0 Apr16 ? 00:00:00 [nfsiod]
> > > root 2440 2 0 Apr16 ? 00:00:00 [nfsd4]
> > > root 2441 2 0 Apr16 ? 00:00:00 [nfsd4_callbacks]
> > > root 2442 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2443 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2444 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2445 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2446 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2447 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2448 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2449 2 0 Apr16 ? 00:00:00 [nfsd]
> > > root 2667 2 0 Apr16 ? 00:00:00 [nfsv4.0-svc]
> > > jocke 27048 26888 0 14:28 pts/3 00:00:00 grep --colour=auto nfs
> > >
> > > Got a new pcap file also:
> > > http://ftp-us.transmode.se/get/?id=6f935e1d7e105d01e9a5b907c6493521
> nfs5.pcap
> > >
> > > The load is not that noticeable so I can stay in this mode a while,
> until I go
> > > home today.
> >
> > So left it overnight and this morning my NFS client had completely
> looked up,
> > had to press the power button. This has happened twice now.
> >
> > One more piece of info, we think this problem started when NFS server
> > was upgraded from 3.4.28 to 3.4.39
> >
> > I have no idea how to move forward now. Trond, are you also stuck?
> >
> > Jocke
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@netapp.com
www.netapp.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-23 13:52 ` Myklebust, Trond
@ 2013-04-23 14:14 ` Joakim Tjernlund
2013-04-23 14:18 ` Myklebust, Trond
[not found] ` <OF940BEB3A.13ADB6E5-ONC1257B56.004D412D-C1257B56.004E3C54@tran <1366726687.35524.6.camel@leira.trondhjem.org>
1 sibling, 1 reply; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-23 14:14 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
15:52:06:
>
> On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > So, it happened again. Just when hitting search on bugs.gentoo.org in
> > firefox 17.0.3
> >
> > This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over and
over
> > again and FF was hung. Not posting the logs as it does not appear to
> > do any good. Nothing in dmesg either.
> >
> > Noticed this patch on the NFS list:
> > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > I wonder if that could be a potential cure and if so, could it be
> > backported to 3.4?
>
> It is in the testing branch on
>
> http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
>
> if you want to try it out. I'm not planning on backporting anything that
> hasn't been labelled with a Cc: stable in that branch.
Well, we won't use tip of linus tree in production so there is
little point to use your testing branch. However it looks like a trivial
backport so I can test it on my client easily.
Even the NFS server if required, is the above referenced patch for
NFS client/server or both? Any chance this is the culprit?
Jocke
PS.
I guess I should throw in
NFSv4: Ensure the LOCK call cannot use the delegation stateid
too?
>
> Cheers
> Trond
>
> > Jocke
> >
> > Joakim Tjernlund/Transmode wrote on 2013/04/19 12:54:38:
> > >
> > > Joakim Tjernlund/Transmode wrote on 2013/04/18 14:34:03:
> > > >
> > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
2013/04/17
> > 00:06:51:
> > > > >
> > > > > On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
> > 2013/04/16
> > > > > > 17:36:55:
> > > > > >
> > > > > > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > > > > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > > > > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > > > > > Date: 2013/04/16 17:37
> > > > > > > Subject: Re: NFS loop on 3.4.39
> > > > > > >
> > > > > > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > > > > > Here we go again, this time i happened while browsing the
> > Boston news
> > > > > > on
> > > > > > > > www.dn.se
> > > > > > > > Now gvfsd-metadata is turned off(not running at all) and I
> > get:
> > > > > > > > 10:28:44.616146 IP 192.168.201.44.nfs >
> > 172.20.4.10.3671768838: reply
> > > > > > ok
> > > > > > > > 52 getattr ERROR: unk 10024
> > > > > > >
> > > > > > > Part of the reason why you are getting no response to these
> > posts is
> > > > > > > that you are posting tcpdump-decoded data. Tcpdump still has
no
> > support
> > > > > > > for NFSv4, and therefore completely garbles the output by
trying
> > to
> > > > > > > interpret it as NFSv2/v3.
> > > > > > > In general, if you are posting network traffic, please
record it
> > as
> > > > > > > binary raw packet data (using the '-w' option on tcdump) so
that
> > we can
> > > > > > > look at the full contents. Either include it as an
attachment,
> > or
> > > > > > > provide us with details on how to download it from an http
> > server.
> > > > > > >
> > > > > > > Other information that is needed in order to make sense of
NFS
> > bug
> > > > > > > reports includes:
> > > > > >
> > > > > > Thank you Trond, I figured there was something missing but I
> > didn't know
> > > > > > where to start but here goes:
> > > > > >
> > > > > > >
> > > > > > > - client OS (non-linux) or kernel version (linux)
> > > > > > Client OS Linux 3.4.39, x86
> > > > > >
> > > > > > > - mount options on the client
> > > > > > ~ # ypmatch jocke auto.home
> > > > > > -fstype=nfs,soft devsrv:/mnt/home/jocke
> > > > > >
> > > > > > > - server OS (non-linux) or kernel version (linux)
> > > > > > Server OS Linux 3.4.39, amd64
> > > > > >
> > > > > > > - type of exported filesystem on the server
> > > > > > XFS
> > > > > >
> > > > > > > - contents of /etc/exports on the server
> > > > > > more /etc/exports
> > > > > > # /etc/exports: NFS file systems being exported. See
exports(5).
> > > > > > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > > > > > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > > > > > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > > > > > /tftproot *(rw,async,root_squash,no_subtree_check)
> > > > > > /mnt/images
*(rw,async,no_root_squash,no_subtree_check,insecure)
> > > > > > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> > > > > >
> > > > > > /mnt/home is the one failing
> > > > > >
> > > > > > >
> > > > > > > Please ensure that you always include those in your emails.
> > > > > >
> > > > > > nfs.pcap:
> > > > > >
> > http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> > > > > >
> > > > > > nfs2.pcap:
> > > > > >
> > http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> > > > > >
> > > > > > nfs3.pcap:
> > > > > >
> > http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> > > > > >
> > > > > > nfs4.pcap:
> > > > > >
> > http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> > > > > >
> > > > > > nfs3.pcap is the gvsd-metadata problem one can find using
google,
> > doesn't
> > > > > > have to be a NFS problem
> > > > > > The other 3 all come from surfing the www using firefox 17.0.3
> > > > >
> > > > > The nfs2.pcap file and nfs4.pcap seem to show the server
returning
> > > > > NFS4ERR_OLD_STATEID, which usually means that the client has an
> > > > > OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server
has
> > > > > updated the stateid, the client has not yet received the reply.
The
> > > > > problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
> > > > >
> > > > > The nfs.pcap file is resending a load of LOCK requests that are
> > > > > receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the
> > recovery
> > > > > engine to kick in and try to recover the OPEN.
> > > > >
> > > > > So when you do 'ps -efwww', on any of these clients, do you see
a
> > > > > process with a name containing the server IP address
> > (192.168.201.44)?
> > > > >
> > > > > Also, is there anything special in the log when you do 'dmesg -s
> > 90000'?
> >
> > > > Of course this happened again while I wasn't looking so I don't
know
> > what
> > > > caused it, probably firefox though.
> > > >
> > > > There is nothing in dmesg and ps -efwww has no hit on IP
> > > > address 192.168.201.44, the closest I can get is:
> > > > ps -efwww | grep nfs
> > > > root 568 2 0 Apr16 ? 00:00:00 [nfsiod]
> > > > root 2440 2 0 Apr16 ? 00:00:00 [nfsd4]
> > > > root 2441 2 0 Apr16 ? 00:00:00 [nfsd4_callbacks]
> > > > root 2442 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2443 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2444 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2445 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2446 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2447 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2448 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2449 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > root 2667 2 0 Apr16 ? 00:00:00 [nfsv4.0-svc]
> > > > jocke 27048 26888 0 14:28 pts/3 00:00:00 grep --colour=auto
nfs
> > > >
> > > > Got a new pcap file also:
> > > >
http://ftp-us.transmode.se/get/?id=6f935e1d7e105d01e9a5b907c6493521
> > nfs5.pcap
> > > >
> > > > The load is not that noticeable so I can stay in this mode a
while,
> > until I go
> > > > home today.
> > >
> > > So left it overnight and this morning my NFS client had completely
> > looked up,
> > > had to press the power button. This has happened twice now.
> > >
> > > One more piece of info, we think this problem started when NFS
server
> > > was upgraded from 3.4.28 to 3.4.39
> > >
> > > I have no idea how to move forward now. Trond, are you also stuck?
> > >
> > > Jocke
>
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-23 14:14 ` Joakim Tjernlund
@ 2013-04-23 14:18 ` Myklebust, Trond
2013-04-23 14:34 ` Joakim Tjernlund
2013-04-24 13:16 ` Joakim Tjernlund
0 siblings, 2 replies; 21+ messages in thread
From: Myklebust, Trond @ 2013-04-23 14:18 UTC (permalink / raw)
To: Joakim Tjernlund; +Cc: linux-nfs
On Tue, 2013-04-23 at 16:14 +0200, Joakim Tjernlund wrote:
> "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
> 15:52:06:
> >
> > On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > > So, it happened again. Just when hitting search on bugs.gentoo.org in
> > > firefox 17.0.3
> > >
> > > This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over and
> over
> > > again and FF was hung. Not posting the logs as it does not appear to
> > > do any good. Nothing in dmesg either.
> > >
> > > Noticed this patch on the NFS list:
> > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > > I wonder if that could be a potential cure and if so, could it be
> > > backported to 3.4?
> >
> > It is in the testing branch on
> >
> > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
> >
> > if you want to try it out. I'm not planning on backporting anything that
> > hasn't been labelled with a Cc: stable in that branch.
>
> Well, we won't use tip of linus tree in production so there is
> little point to use your testing branch. However it looks like a trivial
> backport so I can test it on my client easily.
The point of testing would not be to discover if you can use Linus' tree
in production, but rather to see if the problem is already fixed
upstream. If it is, we can bisect to figure out which patch is the fix.
> Even the NFS server if required, is the above referenced patch for
> NFS client/server or both? Any chance this is the culprit?
That's a client patch.
> Jocke
>
> PS.
> I guess I should throw in
> NFSv4: Ensure the LOCK call cannot use the delegation stateid
> too?
> >
> > Cheers
> > Trond
> >
> > > Jocke
> > >
> > > Joakim Tjernlund/Transmode wrote on 2013/04/19 12:54:38:
> > > >
> > > > Joakim Tjernlund/Transmode wrote on 2013/04/18 14:34:03:
> > > > >
> > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
> 2013/04/17
> > > 00:06:51:
> > > > > >
> > > > > > On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > > > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
> > > 2013/04/16
> > > > > > > 17:36:55:
> > > > > > >
> > > > > > > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > > > > > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > > > > > > Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
> > > > > > > > Date: 2013/04/16 17:37
> > > > > > > > Subject: Re: NFS loop on 3.4.39
> > > > > > > >
> > > > > > > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund wrote:
> > > > > > > > > Here we go again, this time i happened while browsing the
> > > Boston news
> > > > > > > on
> > > > > > > > > www.dn.se
> > > > > > > > > Now gvfsd-metadata is turned off(not running at all) and I
>
> > > get:
> > > > > > > > > 10:28:44.616146 IP 192.168.201.44.nfs >
> > > 172.20.4.10.3671768838: reply
> > > > > > > ok
> > > > > > > > > 52 getattr ERROR: unk 10024
> > > > > > > >
> > > > > > > > Part of the reason why you are getting no response to these
> > > posts is
> > > > > > > > that you are posting tcpdump-decoded data. Tcpdump still has
> no
> > > support
> > > > > > > > for NFSv4, and therefore completely garbles the output by
> trying
> > > to
> > > > > > > > interpret it as NFSv2/v3.
> > > > > > > > In general, if you are posting network traffic, please
> record it
> > > as
> > > > > > > > binary raw packet data (using the '-w' option on tcdump) so
> that
> > > we can
> > > > > > > > look at the full contents. Either include it as an
> attachment,
> > > or
> > > > > > > > provide us with details on how to download it from an http
> > > server.
> > > > > > > >
> > > > > > > > Other information that is needed in order to make sense of
> NFS
> > > bug
> > > > > > > > reports includes:
> > > > > > >
> > > > > > > Thank you Trond, I figured there was something missing but I
> > > didn't know
> > > > > > > where to start but here goes:
> > > > > > >
> > > > > > > >
> > > > > > > > - client OS (non-linux) or kernel version (linux)
> > > > > > > Client OS Linux 3.4.39, x86
> > > > > > >
> > > > > > > > - mount options on the client
> > > > > > > ~ # ypmatch jocke auto.home
> > > > > > > -fstype=nfs,soft devsrv:/mnt/home/jocke
> > > > > > >
> > > > > > > > - server OS (non-linux) or kernel version (linux)
> > > > > > > Server OS Linux 3.4.39, amd64
> > > > > > >
> > > > > > > > - type of exported filesystem on the server
> > > > > > > XFS
> > > > > > >
> > > > > > > > - contents of /etc/exports on the server
> > > > > > > more /etc/exports
> > > > > > > # /etc/exports: NFS file systems being exported. See
> exports(5).
> > > > > > > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > > > > > > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > > > > > > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > > > > > > /tftproot *(rw,async,root_squash,no_subtree_check)
> > > > > > > /mnt/images
> *(rw,async,no_root_squash,no_subtree_check,insecure)
> > > > > > > /rescue *(ro,async,no_root_squash,no_subtree_check,insecure)
> > > > > > >
> > > > > > > /mnt/home is the one failing
> > > > > > >
> > > > > > > >
> > > > > > > > Please ensure that you always include those in your emails.
> > > > > > >
> > > > > > > nfs.pcap:
> > > > > > >
> > > http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> > > > > > >
> > > > > > > nfs2.pcap:
> > > > > > >
> > > http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> > > > > > >
> > > > > > > nfs3.pcap:
> > > > > > >
> > > http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> > > > > > >
> > > > > > > nfs4.pcap:
> > > > > > >
> > > http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> > > > > > >
> > > > > > > nfs3.pcap is the gvsd-metadata problem one can find using
> google,
> > > doesn't
> > > > > > > have to be a NFS problem
> > > > > > > The other 3 all come from surfing the www using firefox 17.0.3
> > > > > >
> > > > > > The nfs2.pcap file and nfs4.pcap seem to show the server
> returning
> > > > > > NFS4ERR_OLD_STATEID, which usually means that the client has an
> > > > > > OPEN/CLOSE/LOCK or LOCKU... in flight and that while the server
> has
> > > > > > updated the stateid, the client has not yet received the reply.
> The
> > > > > > problem is that I see no sign of the OPEN/CLOSE/LOCK/LOCKU...
> > > > > >
> > > > > > The nfs.pcap file is resending a load of LOCK requests that are
> > > > > > receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect the
> > > recovery
> > > > > > engine to kick in and try to recover the OPEN.
> > > > > >
> > > > > > So when you do 'ps -efwww', on any of these clients, do you see
> a
> > > > > > process with a name containing the server IP address
> > > (192.168.201.44)?
> > > > > >
> > > > > > Also, is there anything special in the log when you do 'dmesg -s
>
> > > 90000'?
> > >
> > > > > Of course this happened again while I wasn't looking so I don't
> know
> > > what
> > > > > caused it, probably firefox though.
> > > > >
> > > > > There is nothing in dmesg and ps -efwww has no hit on IP
> > > > > address 192.168.201.44, the closest I can get is:
> > > > > ps -efwww | grep nfs
> > > > > root 568 2 0 Apr16 ? 00:00:00 [nfsiod]
> > > > > root 2440 2 0 Apr16 ? 00:00:00 [nfsd4]
> > > > > root 2441 2 0 Apr16 ? 00:00:00 [nfsd4_callbacks]
> > > > > root 2442 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2443 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2444 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2445 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2446 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2447 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2448 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2449 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > root 2667 2 0 Apr16 ? 00:00:00 [nfsv4.0-svc]
> > > > > jocke 27048 26888 0 14:28 pts/3 00:00:00 grep --colour=auto
> nfs
> > > > >
> > > > > Got a new pcap file also:
> > > > >
> http://ftp-us.transmode.se/get/?id=6f935e1d7e105d01e9a5b907c6493521
> > > nfs5.pcap
> > > > >
> > > > > The load is not that noticeable so I can stay in this mode a
> while,
> > > until I go
> > > > > home today.
> > > >
> > > > So left it overnight and this morning my NFS client had completely
> > > looked up,
> > > > had to press the power button. This has happened twice now.
> > > >
> > > > One more piece of info, we think this problem started when NFS
> server
> > > > was upgraded from 3.4.28 to 3.4.39
> > > >
> > > > I have no idea how to move forward now. Trond, are you also stuck?
> > > >
> > > > Jocke
> >
> >
> > --
> > Trond Myklebust
> > Linux NFS client maintainer
> >
> > NetApp
> > Trond.Myklebust@netapp.com
> > www.netapp.com
>
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@netapp.com
www.netapp.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-23 14:18 ` Myklebust, Trond
@ 2013-04-23 14:34 ` Joakim Tjernlund
2013-04-24 13:16 ` Joakim Tjernlund
1 sibling, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-23 14:34 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
16:18:07:
>
> On Tue, 2013-04-23 at 16:14 +0200, Joakim Tjernlund wrote:
> > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
> > 15:52:06:
> > >
> > > On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > > > So, it happened again. Just when hitting search on bugs.gentoo.org
in
> > > > firefox 17.0.3
> > > >
> > > > This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over
and
> > over
> > > > again and FF was hung. Not posting the logs as it does not appear
to
> > > > do any good. Nothing in dmesg either.
> > > >
> > > > Noticed this patch on the NFS list:
> > > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > > > I wonder if that could be a potential cure and if so, could it be
> > > > backported to 3.4?
> > >
> > > It is in the testing branch on
> > >
> > > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
> > >
> > > if you want to try it out. I'm not planning on backporting anything
that
> > > hasn't been labelled with a Cc: stable in that branch.
> >
> > Well, we won't use tip of linus tree in production so there is
> > little point to use your testing branch. However it looks like a
trivial
> > backport so I can test it on my client easily.
hmm, after testing a patched 3.4 kernel I could possibly try Linus tree
on my client but I doubt I will have time to bisect it as it
can take days to reproduce. Will have it in mind though.
>
> The point of testing would not be to discover if you can use Linus' tree
> in production, but rather to see if the problem is already fixed
> upstream. If it is, we can bisect to figure out which patch is the fix.
>
> > Even the NFS server if required, is the above referenced patch for
> > NFS client/server or both? Any chance this is the culprit?
>
> That's a client patch.
Thanks, rebuilding my clients kernel now.
>
> > Jocke
> >
> > PS.
> > I guess I should throw in
> > NFSv4: Ensure the LOCK call cannot use the delegation stateid
> > too?
> > >
> > > Cheers
> > > Trond
> > >
> > > > Jocke
> > > >
> > > > Joakim Tjernlund/Transmode wrote on 2013/04/19 12:54:38:
> > > > >
> > > > > Joakim Tjernlund/Transmode wrote on 2013/04/18 14:34:03:
> > > > > >
> > > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
> > 2013/04/17
> > > > 00:06:51:
> > > > > > >
> > > > > > > On Tue, 2013-04-16 at 21:07 +0200, Joakim Tjernlund wrote:
> > > > > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
> > > > 2013/04/16
> > > > > > > > 17:36:55:
> > > > > > > >
> > > > > > > > > From: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
> > > > > > > > > To: Joakim Tjernlund <joakim.tjernlund@transmode.se>,
> > > > > > > > > Cc: "linux-nfs@vger.kernel.org"
<linux-nfs@vger.kernel.org>
> > > > > > > > > Date: 2013/04/16 17:37
> > > > > > > > > Subject: Re: NFS loop on 3.4.39
> > > > > > > > >
> > > > > > > > > On Tue, 2013-04-16 at 12:41 +0200, Joakim Tjernlund
wrote:
> > > > > > > > > > Here we go again, this time i happened while browsing
the
> > > > Boston news
> > > > > > > > on
> > > > > > > > > > www.dn.se
> > > > > > > > > > Now gvfsd-metadata is turned off(not running at all)
and I
> >
> > > > get:
> > > > > > > > > > 10:28:44.616146 IP 192.168.201.44.nfs >
> > > > 172.20.4.10.3671768838: reply
> > > > > > > > ok
> > > > > > > > > > 52 getattr ERROR: unk 10024
> > > > > > > > >
> > > > > > > > > Part of the reason why you are getting no response to
these
> > > > posts is
> > > > > > > > > that you are posting tcpdump-decoded data. Tcpdump still
has
> > no
> > > > support
> > > > > > > > > for NFSv4, and therefore completely garbles the output
by
> > trying
> > > > to
> > > > > > > > > interpret it as NFSv2/v3.
> > > > > > > > > In general, if you are posting network traffic, please
> > record it
> > > > as
> > > > > > > > > binary raw packet data (using the '-w' option on tcdump)
so
> > that
> > > > we can
> > > > > > > > > look at the full contents. Either include it as an
> > attachment,
> > > > or
> > > > > > > > > provide us with details on how to download it from an
http
> > > > server.
> > > > > > > > >
> > > > > > > > > Other information that is needed in order to make sense
of
> > NFS
> > > > bug
> > > > > > > > > reports includes:
> > > > > > > >
> > > > > > > > Thank you Trond, I figured there was something missing but
I
> > > > didn't know
> > > > > > > > where to start but here goes:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > - client OS (non-linux) or kernel version (linux)
> > > > > > > > Client OS Linux 3.4.39, x86
> > > > > > > >
> > > > > > > > > - mount options on the client
> > > > > > > > ~ # ypmatch jocke auto.home
> > > > > > > > -fstype=nfs,soft devsrv:/mnt/home/jocke
> > > > > > > >
> > > > > > > > > - server OS (non-linux) or kernel version (linux)
> > > > > > > > Server OS Linux 3.4.39, amd64
> > > > > > > >
> > > > > > > > > - type of exported filesystem on the server
> > > > > > > > XFS
> > > > > > > >
> > > > > > > > > - contents of /etc/exports on the server
> > > > > > > > more /etc/exports
> > > > > > > > # /etc/exports: NFS file systems being exported. See
> > exports(5).
> > > > > > > > /mnt/home *(rw,async,root_squash,no_subtree_check)
> > > > > > > > /mnt/systemtest *(rw,sync,root_squash,no_subtree_check)
> > > > > > > > /mnt/TNM *(rw,sync,root_squash,no_subtree_check)
> > > > > > > > /tftproot *(rw,async,root_squash,no_subtree_check)
> > > > > > > > /mnt/images
> > *(rw,async,no_root_squash,no_subtree_check,insecure)
> > > > > > > > /rescue
*(ro,async,no_root_squash,no_subtree_check,insecure)
> > > > > > > >
> > > > > > > > /mnt/home is the one failing
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Please ensure that you always include those in your
emails.
> > > > > > > >
> > > > > > > > nfs.pcap:
> > > > > > > >
> > > >
http://ftp-us.transmode.se/get/?id=1bf2561ed2e7d4e379b2936319c82c25
> > > > > > > >
> > > > > > > > nfs2.pcap:
> > > > > > > >
> > > >
http://ftp-us.transmode.se/get/?id=759c7645248a426720da8e9ba7074040
> > > > > > > >
> > > > > > > > nfs3.pcap:
> > > > > > > >
> > > >
http://ftp-us.transmode.se/get/?id=051c6d771978b2407e15e96152bd6e66
> > > > > > > >
> > > > > > > > nfs4.pcap:
> > > > > > > >
> > > >
http://ftp-us.transmode.se/get/?id=5dfab4da6cbbe400697bc1621b541c9f
> > > > > > > >
> > > > > > > > nfs3.pcap is the gvsd-metadata problem one can find using
> > google,
> > > > doesn't
> > > > > > > > have to be a NFS problem
> > > > > > > > The other 3 all come from surfing the www using firefox
17.0.3
> > > > > > >
> > > > > > > The nfs2.pcap file and nfs4.pcap seem to show the server
> > returning
> > > > > > > NFS4ERR_OLD_STATEID, which usually means that the client has
an
> > > > > > > OPEN/CLOSE/LOCK or LOCKU... in flight and that while the
server
> > has
> > > > > > > updated the stateid, the client has not yet received the
reply.
> > The
> > > > > > > problem is that I see no sign of the
OPEN/CLOSE/LOCK/LOCKU...
> > > > > > >
> > > > > > > The nfs.pcap file is resending a load of LOCK requests that
are
> > > > > > > receiving NFS4ERR_BAD_STATEID replies. Normally, I'd expect
the
> > > > recovery
> > > > > > > engine to kick in and try to recover the OPEN.
> > > > > > >
> > > > > > > So when you do 'ps -efwww', on any of these clients, do you
see
> > a
> > > > > > > process with a name containing the server IP address
> > > > (192.168.201.44)?
> > > > > > >
> > > > > > > Also, is there anything special in the log when you do
'dmesg -s
> >
> > > > 90000'?
> > > >
> > > > > > Of course this happened again while I wasn't looking so I
don't
> > know
> > > > what
> > > > > > caused it, probably firefox though.
> > > > > >
> > > > > > There is nothing in dmesg and ps -efwww has no hit on IP
> > > > > > address 192.168.201.44, the closest I can get is:
> > > > > > ps -efwww | grep nfs
> > > > > > root 568 2 0 Apr16 ? 00:00:00 [nfsiod]
> > > > > > root 2440 2 0 Apr16 ? 00:00:00 [nfsd4]
> > > > > > root 2441 2 0 Apr16 ? 00:00:00
[nfsd4_callbacks]
> > > > > > root 2442 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2443 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2444 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2445 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2446 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2447 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2448 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2449 2 0 Apr16 ? 00:00:00 [nfsd]
> > > > > > root 2667 2 0 Apr16 ? 00:00:00 [nfsv4.0-svc]
> > > > > > jocke 27048 26888 0 14:28 pts/3 00:00:00 grep
--colour=auto
> > nfs
> > > > > >
> > > > > > Got a new pcap file also:
> > > > > >
> > http://ftp-us.transmode.se/get/?id=6f935e1d7e105d01e9a5b907c6493521
> > > > nfs5.pcap
> > > > > >
> > > > > > The load is not that noticeable so I can stay in this mode a
> > while,
> > > > until I go
> > > > > > home today.
> > > > >
> > > > > So left it overnight and this morning my NFS client had
completely
> > > > looked up,
> > > > > had to press the power button. This has happened twice now.
> > > > >
> > > > > One more piece of info, we think this problem started when NFS
> > server
> > > > > was upgraded from 3.4.28 to 3.4.39
> > > > >
> > > > > I have no idea how to move forward now. Trond, are you also
stuck?
> > > > >
> > > > > Jocke
> > >
> > >
> > > --
> > > Trond Myklebust
> > > Linux NFS client maintainer
> > >
> > > NetApp
> > > Trond.Myklebust@netapp.com
> > > www.netapp.com
> >
>
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-23 14:18 ` Myklebust, Trond
2013-04-23 14:34 ` Joakim Tjernlund
@ 2013-04-24 13:16 ` Joakim Tjernlund
1 sibling, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-24 13:16 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
16:18:07:
>
> On Tue, 2013-04-23 at 16:14 +0200, Joakim Tjernlund wrote:
> > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
> > 15:52:06:
> > >
> > > On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > > > So, it happened again. Just when hitting search on bugs.gentoo.org
in
> > > > firefox 17.0.3
> > > >
> > > > This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over
and
> > over
> > > > again and FF was hung. Not posting the logs as it does not appear
to
> > > > do any good. Nothing in dmesg either.
> > > >
> > > > Noticed this patch on the NFS list:
> > > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > > > I wonder if that could be a potential cure and if so, could it be
> > > > backported to 3.4?
> > >
> > > It is in the testing branch on
> > >
> > > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
> > >
> > > if you want to try it out. I'm not planning on backporting anything
that
> > > hasn't been labelled with a Cc: stable in that branch.
> >
> > Well, we won't use tip of linus tree in production so there is
> > little point to use your testing branch. However it looks like a
trivial
> > backport so I can test it on my client easily.
>
> The point of testing would not be to discover if you can use Linus' tree
> in production, but rather to see if the problem is already fixed
> upstream. If it is, we can bisect to figure out which patch is the fix.
>
> > Even the NFS server if required, is the above referenced patch for
> > NFS client/server or both? Any chance this is the culprit?
>
> That's a client patch.
Tried 3.4.41+above nfs patch and also 3.8.8, they both have the
NFS loop problem.
Now I am at your
http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary, testing
branch
With any luck the error will show soon.
Question though the loop I see, could it be a NFS server bug ?
If so it does matter what I do on my client I guess.
Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
[not found] ` <OFDF20F4D8.12606C1E-ONC1257B57.00486945-C1257B57.0048EA91@LocalDomain>
@ 2013-04-25 15:31 ` Joakim Tjernlund
2013-04-25 15:59 ` Myklebust, Trond
[not found] ` <OF036BF424.501B6F08 <1366905541.6812.18.camel@leira.trondhjem.org>
1 sibling, 1 reply; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-25 15:31 UTC (permalink / raw)
Cc: Myklebust, Trond, linux-nfs
Joakim Tjernlund/Transmode wrote on 2013/04/24 15:16:26:
>
> "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
16:18:07:
> >
> > On Tue, 2013-04-23 at 16:14 +0200, Joakim Tjernlund wrote:
> > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
> > > 15:52:06:
> > > >
> > > > On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > > > > So, it happened again. Just when hitting search on
bugs.gentoo.org in
> > > > > firefox 17.0.3
> > > > >
> > > > > This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over
and
> > > over
> > > > > again and FF was hung. Not posting the logs as it does not
appear to
> > > > > do any good. Nothing in dmesg either.
> > > > >
> > > > > Noticed this patch on the NFS list:
> > > > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > > > > I wonder if that could be a potential cure and if so, could it
be
> > > > > backported to 3.4?
> > > >
> > > > It is in the testing branch on
> > > >
> > > > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
> > > >
> > > > if you want to try it out. I'm not planning on backporting
anything that
> > > > hasn't been labelled with a Cc: stable in that branch.
> > >
> > > Well, we won't use tip of linus tree in production so there is
> > > little point to use your testing branch. However it looks like a
trivial
> > > backport so I can test it on my client easily.
> >
> > The point of testing would not be to discover if you can use Linus'
tree
> > in production, but rather to see if the problem is already fixed
> > upstream. If it is, we can bisect to figure out which patch is the
fix.
> >
> > > Even the NFS server if required, is the above referenced patch for
> > > NFS client/server or both? Any chance this is the culprit?
> >
> > That's a client patch.
> Tried 3.4.41+above nfs patch and also 3.8.8, they both have the
> NFS loop problem.
>
> Now I am at your
http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary,
> testing branch
> With any luck the error will show soon.
>
> Question though the loop I see, could it be a NFS server bug ?
> If so it does matter what I do on my client I guess.
Ran http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary, testing
branch
for a day without problem.
Then I backed to 3.4.41 +
http://marc.info/?l=linux-nfs&m=136643651710066&w=2 +
http://marc.info/?l=linux-nfs&m=136674349127504&w=2
this morning, been using all day without problem. It is a good start
but not conclusive yet.
Is http://marc.info/?l=linux-nfs&m=136674349127504&w=2 supposed to
fix my type of problem?
Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-25 15:31 ` Joakim Tjernlund
@ 2013-04-25 15:59 ` Myklebust, Trond
2013-04-25 16:12 ` Joakim Tjernlund
0 siblings, 1 reply; 21+ messages in thread
From: Myklebust, Trond @ 2013-04-25 15:59 UTC (permalink / raw)
To: Joakim Tjernlund; +Cc: linux-nfs
On Thu, 2013-04-25 at 17:31 +0200, Joakim Tjernlund wrote:
> Joakim Tjernlund/Transmode wrote on 2013/04/24 15:16:26:
> >
> > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
> 16:18:07:
> > >
> > > On Tue, 2013-04-23 at 16:14 +0200, Joakim Tjernlund wrote:
> > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
> > > > 15:52:06:
> > > > >
> > > > > On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > > > > > So, it happened again. Just when hitting search on
> bugs.gentoo.org in
> > > > > > firefox 17.0.3
> > > > > >
> > > > > > This time I got a NFS loop with NFS4ERR_BAD_STATEID looping over
> and
> > > > over
> > > > > > again and FF was hung. Not posting the logs as it does not
> appear to
> > > > > > do any good. Nothing in dmesg either.
> > > > > >
> > > > > > Noticed this patch on the NFS list:
> > > > > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > > > > > I wonder if that could be a potential cure and if so, could it
> be
> > > > > > backported to 3.4?
> > > > >
> > > > > It is in the testing branch on
> > > > >
> > > > > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
> > > > >
> > > > > if you want to try it out. I'm not planning on backporting
> anything that
> > > > > hasn't been labelled with a Cc: stable in that branch.
> > > >
> > > > Well, we won't use tip of linus tree in production so there is
> > > > little point to use your testing branch. However it looks like a
> trivial
> > > > backport so I can test it on my client easily.
> > >
> > > The point of testing would not be to discover if you can use Linus'
> tree
> > > in production, but rather to see if the problem is already fixed
> > > upstream. If it is, we can bisect to figure out which patch is the
> fix.
> > >
> > > > Even the NFS server if required, is the above referenced patch for
> > > > NFS client/server or both? Any chance this is the culprit?
> > >
> > > That's a client patch.
>
> > Tried 3.4.41+above nfs patch and also 3.8.8, they both have the
> > NFS loop problem.
> >
> > Now I am at your
> http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary,
> > testing branch
> > With any luck the error will show soon.
> >
> > Question though the loop I see, could it be a NFS server bug ?
> > If so it does matter what I do on my client I guess.
>
> Ran http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary, testing
> branch
> for a day without problem.
>
> Then I backed to 3.4.41 +
> http://marc.info/?l=linux-nfs&m=136643651710066&w=2 +
> http://marc.info/?l=linux-nfs&m=136674349127504&w=2
> this morning, been using all day without problem. It is a good start
> but not conclusive yet.
>
> Is http://marc.info/?l=linux-nfs&m=136674349127504&w=2 supposed to
> fix my type of problem?
No. That's a follow up patch to commit
92b40e93849e29f9ca661de6442bb66282738bf7 (NFSv4: Use the open stateid if
the delegation has the wrong mode).
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@netapp.com
www.netapp.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
2013-04-25 15:59 ` Myklebust, Trond
@ 2013-04-25 16:12 ` Joakim Tjernlund
0 siblings, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-25 16:12 UTC (permalink / raw)
To: Myklebust, Trond; +Cc: linux-nfs
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/25
17:59:01:
>
> On Thu, 2013-04-25 at 17:31 +0200, Joakim Tjernlund wrote:
> > Joakim Tjernlund/Transmode wrote on 2013/04/24 15:16:26:
> > >
> > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/23
> > 16:18:07:
> > > >
> > > > On Tue, 2013-04-23 at 16:14 +0200, Joakim Tjernlund wrote:
> > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
2013/04/23
> > > > > 15:52:06:
> > > > > >
> > > > > > On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > > > > > > So, it happened again. Just when hitting search on
> > bugs.gentoo.org in
> > > > > > > firefox 17.0.3
> > > > > > >
> > > > > > > This time I got a NFS loop with NFS4ERR_BAD_STATEID looping
over
> > and
> > > > > over
> > > > > > > again and FF was hung. Not posting the logs as it does not
> > appear to
> > > > > > > do any good. Nothing in dmesg either.
> > > > > > >
> > > > > > > Noticed this patch on the NFS list:
> > > > > > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > > > > > > I wonder if that could be a potential cure and if so, could
it
> > be
> > > > > > > backported to 3.4?
> > > > > >
> > > > > > It is in the testing branch on
> > > > > >
> > > > > > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
> > > > > >
> > > > > > if you want to try it out. I'm not planning on backporting
> > anything that
> > > > > > hasn't been labelled with a Cc: stable in that branch.
> > > > >
> > > > > Well, we won't use tip of linus tree in production so there is
> > > > > little point to use your testing branch. However it looks like a
> > trivial
> > > > > backport so I can test it on my client easily.
> > > >
> > > > The point of testing would not be to discover if you can use
Linus'
> > tree
> > > > in production, but rather to see if the problem is already fixed
> > > > upstream. If it is, we can bisect to figure out which patch is the
> > fix.
> > > >
> > > > > Even the NFS server if required, is the above referenced patch
for
> > > > > NFS client/server or both? Any chance this is the culprit?
> > > >
> > > > That's a client patch.
> >
> > > Tried 3.4.41+above nfs patch and also 3.8.8, they both have the
> > > NFS loop problem.
> > >
> > > Now I am at your
> > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary,
> > > testing branch
> > > With any luck the error will show soon.
> > >
> > > Question though the loop I see, could it be a NFS server bug ?
> > > If so it does matter what I do on my client I guess.
> >
> > Ran http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary,
testing
> > branch
> > for a day without problem.
> >
> > Then I backed to 3.4.41 +
> > http://marc.info/?l=linux-nfs&m=136643651710066&w=2 +
> > http://marc.info/?l=linux-nfs&m=136674349127504&w=2
> > this morning, been using all day without problem. It is a good start
> > but not conclusive yet.
> >
> > Is http://marc.info/?l=linux-nfs&m=136674349127504&w=2 supposed to
> > fix my type of problem?
>
> No. That's a follow up patch to commit
> 92b40e93849e29f9ca661de6442bb66282738bf7 (NFSv4: Use the open stateid if
> the delegation has the wrong mode).
hmm, that commit is the first one I listed,
http://marc.info/?l=linux-nfs&m=136643651710066&w=2
and I know that using only that one does NOT fix the problem. I was hoping
that both of them could
be the answer?
Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: NFS loop on 3.4.39
[not found] ` <OF53E7EB3E.683DA47E-ONC1257B58.0058BE67-C1257B58.005908EA@LocalDomain>
@ 2013-05-17 13:18 ` Joakim Tjernlund
0 siblings, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-05-17 13:18 UTC (permalink / raw)
Cc: Myklebust, Trond, linux-nfs
Back on this old problem now with some more input. Upgraded my client to
3.8.13(server is on 3.4.44)
and got the NFS4ERR_OLD_STATEID ping pong effect.
Firefox was locked with process state "Dl" and I got this dmesg:
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000000
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
INFO: task Gecko_IOThread:10655 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10655 6336 0x00000004
f44a3400 00000086 ece3dea0 00000000 00000000 c10cb15b ef6fe030 c17e4280
f30c17c0 ece3de60 c17e4280 c17e4280 ef6fe030 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434638 f4434638
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task Gecko_IOThread:10691 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10691 6336 0x00000000
f44a3400 00000086 d0247ea0 00000000 00000000 c10cb15b ef317400 c17e4280
f31590c0 d0247e60 c17e4280 c17e4280 ef317400 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434ae8 f4434ae8
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000004
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
INFO: task Gecko_IOThread:10655 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10655 6355 0x00000004
f44a3400 00000086 ece3dea0 00000000 00000000 c10cb15b ef6fe030 c17e4280
f30c17c0 ece3de60 c17e4280 c17e4280 ef6fe030 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434638 f4434638
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task Gecko_IOThread:10691 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10691 6355 0x00000000
f44a3400 00000086 d0247ea0 00000000 00000000 c10cb15b ef317400 c17e4280
f31590c0 d0247e60 c17e4280 c17e4280 ef317400 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434ae8 f4434ae8
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000004
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
INFO: task Gecko_IOThread:10655 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10655 6355 0x00000004
f44a3400 00000086 ece3dea0 00000000 00000000 c10cb15b ef6fe030 c17e4280
f30c17c0 ece3de60 c17e4280 c17e4280 ef6fe030 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434638 f4434638
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
INFO: task Gecko_IOThread:10691 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Gecko_IOThread D 00000000 0 10691 6355 0x00000000
f44a3400 00000086 d0247ea0 00000000 00000000 c10cb15b ef317400 c17e4280
f31590c0 d0247e60 c17e4280 c17e4280 ef317400 c17e4280 c10cb284 00000002
0000000e 00000000 00000000 ffffffff 00000000 00000000 f4434ae8 f4434ae8
Call Trace:
[<c10cb15b>] ? tag_pages_for_writeback+0x5b/0xa0
[<c10cb284>] ? write_cache_pages+0xe4/0x340
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3eae>] ? filemap_fdatawait_range+0xde/0x170
[<c10c42cb>] ? filemap_write_and_wait_range+0x7b/0x90
[<c11ae336>] ? nfs_file_fsync+0x36/0xe0
[<c11ae300>] ? nfs_lock+0x150/0x150
[<c1125458>] ? vfs_fsync+0x48/0x60
[<c10fffee>] ? filp_close+0x2e/0x80
[<c1100094>] ? sys_close+0x54/0xa0
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
nfsd: last server has exited, flushing export cache
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
INFO: task mozStorage #2:6355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mozStorage #2 D f5619280 0 6355 1 0x00000004
f446d440 00000086 f446d65c f5619280 f3745300 f5619280 ef0da810 c17e4280
f3745300 c15a897e c17e4280 c17e4280 ef0da810 c17e4280 ef0da810 c17e4280
f3745300 f3745300 c17e4280 c17e4280 ef0da810 c17e4280 f46124f8 f46124f8
Call Trace:
[<c15a897e>] ? __schedule+0x23e/0x660
[<c1088d70>] ? ktime_get_ts+0xf0/0x120
[<c15a907e>] ? io_schedule+0x6e/0xb0
[<c10c35f5>] ? sleep_on_page+0x5/0x10
[<c15a75b5>] ? __wait_on_bit+0x45/0x70
[<c10c35f0>] ? __lock_page+0x80/0x80
[<c10c383d>] ? wait_on_page_bit+0x8d/0xa0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c10c3a7c>] ? grab_cache_page_write_begin+0x9c/0xe0
[<c106ec10>] ? autoremove_wake_function+0x40/0x40
[<c11aea2b>] ? nfs_write_begin+0x5b/0x1c0
[<c11ae81f>] ? nfs_write_end+0x4f/0x200
[<c10c2e27>] ? generic_file_buffered_write+0xe7/0x230
[<c10c478e>] ? __generic_file_aio_write+0x27e/0x500
[<c15a7fe8>] ? __mutex_lock_slowpath+0x1d8/0x270
[<c10c4a84>] ? generic_file_aio_write+0x74/0xf0
[<c11aecc2>] ? nfs_file_write+0x92/0x1c0
[<c1101a6d>] ? do_sync_write+0xcd/0x110
[<c1052809>] ? kmap_atomic_prot+0xd9/0x100
[<c11021fa>] ? rw_verify_area+0x6a/0x130
[<c11019a0>] ? wait_on_retry_sync_kiocb+0x50/0x50
[<c1102560>] ? vfs_write+0xa0/0x140
[<c11018f3>] ? vfs_llseek+0x33/0x40
[<c1102801>] ? sys_write+0x41/0x80
[<c15aa4d8>] ? sysenter_do_call+0x12/0x28
[<c15a0000>] ? identify_cpu+0xc0/0x36c
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
Joakim Tjernlund/Transmode wrote on 2013/04/25 18:12:29:
>
> "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on 2013/04/25
17:59:01:
> >
> > On Thu, 2013-04-25 at 17:31 +0200, Joakim Tjernlund wrote:
> > > Joakim Tjernlund/Transmode wrote on 2013/04/24 15:16:26:
> > > >
> > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
2013/04/23
> > > 16:18:07:
> > > > >
> > > > > On Tue, 2013-04-23 at 16:14 +0200, Joakim Tjernlund wrote:
> > > > > > "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote on
2013/04/23
> > > > > > 15:52:06:
> > > > > > >
> > > > > > > On Tue, 2013-04-23 at 15:38 +0200, Joakim Tjernlund wrote:
> > > > > > > > So, it happened again. Just when hitting search on
> > > bugs.gentoo.org in
> > > > > > > > firefox 17.0.3
> > > > > > > >
> > > > > > > > This time I got a NFS loop with NFS4ERR_BAD_STATEID
looping over
> > > and
> > > > > > over
> > > > > > > > again and FF was hung. Not posting the logs as it does not
> > > appear to
> > > > > > > > do any good. Nothing in dmesg either.
> > > > > > > >
> > > > > > > > Noticed this patch on the NFS list:
> > > > > > > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> > > > > > > > I wonder if that could be a potential cure and if so,
could it
> > > be
> > > > > > > > backported to 3.4?
> > > > > > >
> > > > > > > It is in the testing branch on
> > > > > > >
> > > > > > >
http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary
> > > > > > >
> > > > > > > if you want to try it out. I'm not planning on backporting
> > > anything that
> > > > > > > hasn't been labelled with a Cc: stable in that branch.
> > > > > >
> > > > > > Well, we won't use tip of linus tree in production so there is
> > > > > > little point to use your testing branch. However it looks like
a
> > > trivial
> > > > > > backport so I can test it on my client easily.
> > > > >
> > > > > The point of testing would not be to discover if you can use
Linus'
> > > tree
> > > > > in production, but rather to see if the problem is already fixed
> > > > > upstream. If it is, we can bisect to figure out which patch is
the
> > > fix.
> > > > >
> > > > > > Even the NFS server if required, is the above referenced patch
for
> > > > > > NFS client/server or both? Any chance this is the culprit?
> > > > >
> > > > > That's a client patch.
> > >
> > > > Tried 3.4.41+above nfs patch and also 3.8.8, they both have the
> > > > NFS loop problem.
> > > >
> > > > Now I am at your
> > > http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary,
> > > > testing branch
> > > > With any luck the error will show soon.
> > > >
> > > > Question though the loop I see, could it be a NFS server bug ?
> > > > If so it does matter what I do on my client I guess.
> > >
> > > Ran http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=summary,
testing
> > > branch
> > > for a day without problem.
> > >
> > > Then I backed to 3.4.41 +
> > > http://marc.info/?l=linux-nfs&m=136643651710066&w=2 +
> > > http://marc.info/?l=linux-nfs&m=136674349127504&w=2
> > > this morning, been using all day without problem. It is a good start
> > > but not conclusive yet.
> > >
> > > Is http://marc.info/?l=linux-nfs&m=136674349127504&w=2 supposed to
> > > fix my type of problem?
> >
> > No. That's a follow up patch to commit
> > 92b40e93849e29f9ca661de6442bb66282738bf7 (NFSv4: Use the open stateid
if
> > the delegation has the wrong mode).
> hmm, that commit is the first one I listed,
http://marc.info/?l=linux-nfs&m=136643651710066&w=2
> and I know that using only that one does NOT fix the problem. I was
hoping that both of them could
> be the answer?
>
> Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
* NFS loop on 3.4.39
@ 2013-04-15 8:19 Joakim Tjernlund
0 siblings, 0 replies; 21+ messages in thread
From: Joakim Tjernlund @ 2013-04-15 8:19 UTC (permalink / raw)
To: linux-nfs
I get som NFS loop which generates lots of (from tcpdump):
9:48:43.156252 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8406616, win 32885, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.156258 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8426888, win 32885, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.156669 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8445712, win 32885, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.156675 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8459880, win 32883, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.156679 IP 192.168.201.44.nfs > 172.20.4.10.2889942938: reply ok
52 getattr ERROR: unk 10024
09:48:43.156704 IP 172.20.4.10.2923497370 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
09:48:43.156712 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8462776:8467120, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156719 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8467120:8471464, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156725 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8471464:8475808, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156731 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8475808:8480152, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156745 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8480152:8484496, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156748 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8484496:8487392, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 2896
09:48:43.156751 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8487392:8491736, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156755 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8491736:8496080, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156758 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8496080:8500424, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156764 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8500424:8504768, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156767 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8504768:8509112, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156770 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8509112:8512008, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 2896
09:48:43.156773 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8512008:8516352, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156778 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8516352:8520696, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156956 IP 192.168.201.44.nfs > 172.20.4.10.2906720154: reply ok
52 getattr ERROR: unk 10024
09:48:43.156969 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8520696:8525040, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156975 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8525040:8529384, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156980 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8529384:8533728, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156984 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8533728:8536624, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 2896
09:48:43.156988 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8536624:8540968, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156994 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8540968:8545312, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.156999 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8545312:8549656, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.157209 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8464224, win 32872, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.157219 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8549656:8554000, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.157225 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8554000:8558344, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.157230 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [P.], seq
8558344:8558424, ack 9577, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 80
09:48:43.157235 IP 172.20.4.10.2940274586 > 192.168.201.44.nfs: 532
getattr fh 0,0/22
09:48:43.157460 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8485944, win 32885, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.157709 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8504768, win 32885, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.157715 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8526488, win 32885, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.158126 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8543864, win 32885, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.158133 IP 192.168.201.44.nfs > 172.20.4.10.con: Flags [.], ack
8558960, win 32883, options [nop,nop,TS val 43795665 ecr 57419706], length
0
09:48:43.158138 IP 192.168.201.44.nfs > 172.20.4.10.2923497370: reply ok
52 getattr ERROR: unk 10024
09:48:43.158143 IP 192.168.201.44.nfs > 172.20.4.10.2940274586: reply ok
52 getattr ERROR: unk 10024
09:48:43.158149 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], ack
9745, win 23907, options [nop,nop,TS val 57419706 ecr 43795665], length 0
09:48:43.158174 IP 172.20.4.10.2957051802 > 192.168.201.44.nfs: 2892
getattr fh 0,0/22
09:48:43.158192 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8561856:8566200, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.158197 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8566200:8570544, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.158204 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8570544:8574888, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.158210 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8574888:8579232, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.158216 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8579232:8583576, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.158222 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8583576:8586472, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 2896
09:48:43.158226 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8586472:8590816, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.158233 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8590816:8595160, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
09:48:43.158239 IP 172.20.4.10.con > 192.168.201.44.nfs: Flags [.], seq
8595160:8599504, ack 9745, win 23907, options [nop,nop,TS val 57419706 ecr
43795665], length 4344
I wonder if this is a variant on:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/nfs/nfs4proc.c?h=linux-3.6.y&id=9fa2b82e5592a7aa7a63b7f6a32c5aa9e580643a
which does seem to be in the 3.4 stable branch?
Jocke
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2013-05-17 13:18 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <OF6F031A84.8897519B-ONC1257B4E.002D1BA3-C1257B4E.002DBED0@LocalDomain>
2013-04-15 8:41 ` NFS loop on 3.4.39 Joakim Tjernlund
[not found] ` <OF83CD52DB.8DBE497B-ONC1257B4E.002F8695-C1257B4E.002FBBD0@LocalDomain>
2013-04-15 8:48 ` Joakim Tjernlund
[not found] ` <OF64305930.0F7881A0-ONC1257B4E.0030250F-C1257B4E.00306CEC@LocalDomain>
2013-04-16 10:41 ` Joakim Tjernlund
2013-04-16 15:36 ` Myklebust, Trond
2013-04-16 19:07 ` Joakim Tjernlund
2013-04-16 22:06 ` Myklebust, Trond
2013-04-16 22:43 ` Joakim Tjernlund
2013-04-17 12:03 ` Joakim Tjernlund
2013-04-18 12:34 ` Joakim Tjernlund
[not found] ` <OF97D68E61.A3C35530-ONC1257B51.0043E567-C1257B51.00450958@LocalDomain>
2013-04-19 10:54 ` Joakim Tjernlund
[not found] ` <OFF8AA22C1.D96F08FD-ONC1257B52.003B6FD9-C1257B52.003BEEF9@LocalDomain>
2013-04-23 13:38 ` Joakim Tjernlund
2013-04-23 13:52 ` Myklebust, Trond
2013-04-23 14:14 ` Joakim Tjernlund
2013-04-23 14:18 ` Myklebust, Trond
2013-04-23 14:34 ` Joakim Tjernlund
2013-04-24 13:16 ` Joakim Tjernlund
[not found] ` <OF940BEB3A.13ADB6E5-ONC1257B56.004D412D-C1257B56.004E3C54@tran <1366726687.35524.6.camel@leira.trondhjem.org>
[not found] ` <OFDF20F4D8.12606C1E-ONC1257B57.00486945-C1257B57.0048EA91@LocalDomain>
2013-04-25 15:31 ` Joakim Tjernlund
2013-04-25 15:59 ` Myklebust, Trond
2013-04-25 16:12 ` Joakim Tjernlund
[not found] ` <OF036BF424.501B6F08 <1366905541.6812.18.camel@leira.trondhjem.org>
[not found] ` <OF53E7EB3E.683DA47E-ONC1257B58.0058BE67-C1257B58.005908EA@LocalDomain>
2013-05-17 13:18 ` Joakim Tjernlund
2013-04-15 8:19 Joakim Tjernlund
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.