* NFS problems with Linux-2.4 @ 2003-05-13 14:50 jlnance 2003-05-13 15:19 ` Trond Myklebust 2003-05-13 19:07 ` jjs 0 siblings, 2 replies; 12+ messages in thread From: jlnance @ 2003-05-13 14:50 UTC (permalink / raw) To: trond.myklebust, linux-kernel; +Cc: gary.nifong, jlnance, david.thomas Hello all, I am having some problems with NFS which I suspect may be a bug in the 2.4 kernels. I can probably come up with a small testcase, but before I do that I would like to describe the problem and see if it is something that is susposed to work. Perhaps I simply do not understand the guarantees that NFS makes. The setup is like this. I have two machines which share an NFS mounted directory. The NFS server is a network appliance box. Machine A does an fopen/fwrite/fclose to create a file on the NFS filesystem. It then sends a message to machine B. Machine B then attemps to fopen the file, but fopen fails (as does stat). If I add code that sleeps for a couple of seconds and retries the fopen then everything works. I have seen the problem on both IA64 machines running the kernel 2.4.18 from Red Hats Advanced Server and on x86 machines running Red Hats 2.4.7-10smp kernel. I have not tried other linux kernels (I am not root), but I have run the same program under Solaris (sparc) and have never observed this. The IA64 and x86 machines were on different networks and using different network appliance servers. The IA64 /proc/mounts entry is: na1:/vol/h1/home /remote/na1h1home nfs rw,v3,rsize=4096,wsize=4096,hard,intr,udp,lock,addr=na1 0 0 and the x86 entry is: na1-rtp:/vol/vol0/home/jlnance /home/jlnance nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=na1-rtp 0 0 If you would like more information, please let me know. Thanks, Jim ^ permalink raw reply [flat|nested] 12+ messages in thread
* NFS problems with Linux-2.4 2003-05-13 14:50 NFS problems with Linux-2.4 jlnance @ 2003-05-13 15:19 ` Trond Myklebust 2003-05-15 15:22 ` Jim Nance 2003-05-13 19:07 ` jjs 1 sibling, 1 reply; 12+ messages in thread From: Trond Myklebust @ 2003-05-13 15:19 UTC (permalink / raw) To: jlnance; +Cc: linux-kernel, gary.nifong, jlnance, david.thomas Could you please try with a newer kernel. The close-to-open cache consistency fixes are a relatively recent addition to the Linux NFS client. I dunno if RedHat's 2.4.18 kernel has them. 2.4.7 certainly does not. Cheers, Trond ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-13 15:19 ` Trond Myklebust @ 2003-05-15 15:22 ` Jim Nance 2003-05-18 15:00 ` Trond Myklebust 0 siblings, 1 reply; 12+ messages in thread From: Jim Nance @ 2003-05-15 15:22 UTC (permalink / raw) To: Trond Myklebust Cc: jlnance, linux-kernel, gary.nifong, James.Nance, david.thomas [-- Attachment #1: Type: text/plain, Size: 1035 bytes --] On Tue, May 13, 2003 at 05:19:23PM +0200, Trond Myklebust wrote: > > Could you please try with a newer kernel. The close-to-open cache > consistency fixes are a relatively recent addition to the Linux NFS > client. I dunno if RedHat's 2.4.18 kernel has them. > > 2.4.7 certainly does not. I tried again with the 2.4.20 based kernel that Red Hat released yesterday (2.4.20-13.7bigmem). The problem that I was seeing occurs less frequently there, but it still happens. I have attached a program which can reproduce this. If you run it under 2.4.7 it fails instantly. If you use 2.4.20 it may take a minute or so but it will also fail. Thanks, Jim PS: Do you know if there is any way to work around this problem from within my program? -- ---------------------------------------------------------------------------- Jim Nance Synopsys (919) 425-7219 Do you have sweet iced tea? jlnance at synopsys.com No, but there's sugar on the table. [-- Attachment #2: p1.c --] [-- Type: text/plain, Size: 4507 bytes --] /* This program demonstrates a problem with the close/open consistency * of NFS file systems under Linux. It fails very rapidy with Red Hats * 2.4.7-10smp kernel. This kernel was known to have bugs. It also fails * with Red Hats 2.4.20-13.7bigmem kernel, which was thought to have this * bug fixed. For my testcase both linux machines were talking to a * network applicance file server and mounted like this: * * na1-rtp:/vol/vol0/home/jlnance /home/jlnance nfs rw,v3,rsize=4096,\ * wsize=4096,hard,intr,udp,lock,addr=na1-rtp 0 0 * * This program needs to be run on 2 machines, assume hostnames A & B. * A and B need to share an NFS mounted file system. * * On machine A: * cd /some/nfs/path/common/to/both * ./p1 s * * On machine B: * cd /some/nfs/path/common/to/both * ./p1 c A * * After a while you may see output similar to: * cayman> ./p1 s * Failed to find #0 which client wrote * Failed on file number 483 */ #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/socket.h> #include <netinet/in.h> #include <fcntl.h> #include <unistd.h> #include <errno.h> #include <netdb.h> #define PORT 12387 #define FLEN 16 void die() { perror(""); exit(-1); } void Write(int fd, char *buff, size_t len) { for(;;) { int nsent=write(fd, buff, len); if(nsent==0) exit(0); if(nsent==-1) { if(errno!=EINTR) die(); } else { buff += nsent; len -= nsent; if(len==0) { return; } } } } void Read(int fd, char *buff, size_t len) { for(;;) { int nread=read(fd, buff, len); if(nread==0) exit(0); if(nread==-1) { if(errno!=EINTR) die(); } else { buff += nread; len -= nread; if(len==0) { return; } } } } int server() { int sock = socket(AF_INET, SOCK_STREAM, 0); if(sock==-1) die(); else { struct sockaddr_in name; int on = 1; name.sin_family = AF_INET; name.sin_addr.s_addr = htonl(INADDR_ANY); name.sin_port = htons(PORT); setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof on); if(bind(sock, (struct sockaddr*)&name, sizeof(name))==-1) die(); else { if(listen(sock, 1)==-1) die(); else { int tsock = accept(sock, 0, 0); if(tsock!=-1) { int cnt; for(cnt=0; cnt<100000; cnt++) { int fd; char dummy; char number[FLEN]; struct stat sbuf; /*sprintf(number, "#%d", cnt);*/ sprintf(number, "#%d", 0); Write(tsock, number, sizeof(number)); Read(tsock, &dummy, 1); if(stat(number, &sbuf)) { fprintf(stderr, "Failed to find %s which client wrote\n", number); fprintf(stderr, "Failed on file number %d\n", cnt); exit(-2); } unlink(number); } } } } } return 0; } int client(char *server) { struct hostent *info = gethostbyname(server); if(!info) die(); else { int rsocket = socket(AF_INET, SOCK_STREAM, 0); if(rsocket==-1) die(); else { struct sockaddr_in name; name.sin_family = AF_INET; name.sin_port = htons(PORT); memcpy(&name.sin_addr, info->h_addr_list[0], sizeof(struct in_addr)); if(connect(rsocket, (struct sockaddr*)&name, sizeof(name))==-1) die(); else { for(;;) { int fd; char fname[FLEN]; char tname[FLEN+8]; Read(rsocket, fname, sizeof(fname)); strcpy(tname, fname); strcat(tname, ".tmp"); fd = open(tname, O_WRONLY|O_CREAT, 0600); if(fd==-1) die(); Write(fd, fname, sizeof(fname)); /* Junk data */ close(fd); rename(tname, fname); Write(rsocket, fname, 1); /* Tells the server we are done */ } } } } return 0; } void usage(char *prog) { fprintf(stderr, "Usage:\n"); fprintf(stderr, " %s s\n", prog); fprintf(stderr, " %s c servername\n", prog); fprintf(stderr, " Run 1 of each in the same NFS directory on 2 different " "machines\n Two processes total\n"); exit(-1); } int main(int ac, char **av) { if(ac<2) { usage(av[0]); } if(av[1][0]=='s') { return server(); }else if(ac<3) { usage(av[0]); } else if(av[1][0]=='c') { return client(av[2]); } else { usage(av[0]); } return -1; } ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-15 15:22 ` Jim Nance @ 2003-05-18 15:00 ` Trond Myklebust 2003-05-19 0:53 ` jlnance 2003-05-19 20:02 ` Jim Nance 0 siblings, 2 replies; 12+ messages in thread From: Trond Myklebust @ 2003-05-18 15:00 UTC (permalink / raw) To: Jim Nance Cc: Trond Myklebust, jlnance, linux-kernel, gary.nifong, James.Nance, david.thomas Sorry. stat doesn't obey close-to-open. It relies on standard attribute caching. close-to-open means "open()" (and only "open()") checks data cache consistency... Cheers, Trond ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-18 15:00 ` Trond Myklebust @ 2003-05-19 0:53 ` jlnance 2003-05-19 11:27 ` Trond Myklebust 2003-05-19 20:02 ` Jim Nance 1 sibling, 1 reply; 12+ messages in thread From: jlnance @ 2003-05-19 0:53 UTC (permalink / raw) To: Trond Myklebust Cc: Jim Nance, jlnance, linux-kernel, gary.nifong, James.Nance, david.thomas On Sun, May 18, 2003 at 05:00:24PM +0200, Trond Myklebust wrote: > > Sorry. stat doesn't obey close-to-open. It relies on standard > attribute caching. close-to-open means "open()" (and only "open()") > checks data cache consistency... Trond, Thanks for the info. Here is a section of the man page for open. Is the information it gives correct wrt using link & stat? O_EXCL When used with O_CREAT, if the file already exists it is an error and the open will fail. In this con text, a symbolic link exists, regardless of where its points to. O_EXCL is broken on NFS file sys tems, programs which rely on it for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also suc cessful. Thanks, Jim ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-19 0:53 ` jlnance @ 2003-05-19 11:27 ` Trond Myklebust 0 siblings, 0 replies; 12+ messages in thread From: Trond Myklebust @ 2003-05-19 11:27 UTC (permalink / raw) To: jlnance; +Cc: Jim Nance, linux-kernel, gary.nifong, James.Nance, david.thomas >>>>> " " == jlnance <jlnance@unity.ncsu.edu> writes: > Thanks for the info. Here is a section of the man page for > open. > Is the information it gives correct wrt using link & stat? Yes. Attempting to link a file will normally update the cached attributes, so stat() will give correct results. Cheers, Trond ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-18 15:00 ` Trond Myklebust 2003-05-19 0:53 ` jlnance @ 2003-05-19 20:02 ` Jim Nance 1 sibling, 0 replies; 12+ messages in thread From: Jim Nance @ 2003-05-19 20:02 UTC (permalink / raw) To: Trond Myklebust Cc: jlnance, linux-kernel, gary.nifong, James.Nance, david.thomas [-- Attachment #1: Type: text/plain, Size: 977 bytes --] On Sun, May 18, 2003 at 05:00:24PM +0200, Trond Myklebust wrote: > > Sorry. stat doesn't obey close-to-open. It relies on standard > attribute caching. close-to-open means "open()" (and only "open()") > checks data cache consistency... Hi Trond, I rewrote my test program so that it uses open() instead of stat(). I also changed it so that it does not rename the file after it writing it. This should only leave close, open, and unlink calls. The program still fails for me after running for a minute or so: cayman> ./p1 s Failed to find #0 which client wrote Failed on file number 10202 Again, this is with 2.4.20 kernel. It fails much faster with a 2.4.7. Thanks, Jim -- ---------------------------------------------------------------------------- Jim Nance Synopsys (919) 425-7219 Do you have sweet iced tea? jlnance@us54.synopsys.com No, but there's sugar on the table. [-- Attachment #2: p1.c --] [-- Type: text/plain, Size: 4809 bytes --] /* This program demonstrates a problem with the close/open consistency * of NFS file systems under Linux. It fails very rapidy with Red Hats * 2.4.7-10smp kernel. This kernel was known to have bugs. It also fails * with Red Hats 2.4.20-13.7bigmem kernel, which was thought to have this * bug fixed. For my testcase both linux machines were talking to a * network applicance file server and mounted like this: * * na1-rtp:/vol/vol0/home/jlnance /home/jlnance nfs rw,v3,rsize=4096,\ * wsize=4096,hard,intr,udp,lock,addr=na1-rtp 0 0 * * This program needs to be run on 2 machines, assume hostnames A & B. * A and B need to share an NFS mounted file system. * * On machine A: * cd /some/nfs/path/common/to/both * ./p1 s * * On machine B: * cd /some/nfs/path/common/to/both * ./p1 c A * * After a while you may see output similar to: * cayman> ./p1 s * Failed to find #0 which client wrote * Failed on file number 483 */ #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/socket.h> #include <netinet/in.h> #include <fcntl.h> #include <unistd.h> #include <errno.h> #include <netdb.h> #define PORT 12387 #define FLEN 16 void die() { perror(""); exit(-1); } void Write(int fd, char *buff, size_t len) { for(;;) { int nsent=write(fd, buff, len); if(nsent==0) exit(0); if(nsent==-1) { if(errno!=EINTR) die(); } else { buff += nsent; len -= nsent; if(len==0) { return; } } } } void Read(int fd, char *buff, size_t len) { for(;;) { int nread=read(fd, buff, len); if(nread==0) exit(0); if(nread==-1) { if(errno!=EINTR) die(); } else { buff += nread; len -= nread; if(len==0) { return; } } } } int server() { int sock = socket(AF_INET, SOCK_STREAM, 0); if(sock==-1) die(); else { struct sockaddr_in name; int on = 1; name.sin_family = AF_INET; name.sin_addr.s_addr = htonl(INADDR_ANY); name.sin_port = htons(PORT); setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof on); if(bind(sock, (struct sockaddr*)&name, sizeof(name))==-1) die(); else { if(listen(sock, 1)==-1) die(); else { int tsock = accept(sock, 0, 0); if(tsock!=-1) { int cnt; for(cnt=0; cnt<100000; cnt++) { int fd; char dummy; char number[FLEN]; struct stat sbuf; /*sprintf(number, "#%d", cnt);*/ sprintf(number, "#%d", 0); Write(tsock, number, sizeof(number)); Read(tsock, &dummy, 1); #if 0 if(stat(number, &sbuf)) { fprintf(stderr, "Failed to find %s which client wrote\n", number); fprintf(stderr, "Failed on file number %d\n", cnt); exit(-2); } #else fd = open(number, O_RDONLY, 0); if(fd<0) { fprintf(stderr, "Failed to find %s which client wrote\n", number); fprintf(stderr, "Failed on file number %d\n", cnt); exit(-2); } #endif close(fd); unlink(number); } } } } } return 0; } int client(char *server) { struct hostent *info = gethostbyname(server); if(!info) die(); else { int rsocket = socket(AF_INET, SOCK_STREAM, 0); if(rsocket==-1) die(); else { struct sockaddr_in name; name.sin_family = AF_INET; name.sin_port = htons(PORT); memcpy(&name.sin_addr, info->h_addr_list[0], sizeof(struct in_addr)); if(connect(rsocket, (struct sockaddr*)&name, sizeof(name))==-1) die(); else { for(;;) { int fd; char fname[FLEN]; char tname[FLEN+8]; Read(rsocket, fname, sizeof(fname)); strcpy(tname, fname); /*strcat(tname, ".tmp");*/ fd = open(tname, O_WRONLY|O_CREAT, 0600); if(fd==-1) die(); Write(fd, fname, sizeof(fname)); /* Junk data */ close(fd); /*rename(tname, fname);*/ Write(rsocket, fname, 1); /* Tells the server we are done */ } } } } return 0; } void usage(char *prog) { fprintf(stderr, "Usage:\n"); fprintf(stderr, " %s s\n", prog); fprintf(stderr, " %s c servername\n", prog); fprintf(stderr, " Run 1 of each in the same NFS directory on 2 different " "machines\n Two processes total\n"); exit(-1); } int main(int ac, char **av) { if(ac<2) { usage(av[0]); } if(av[1][0]=='s') { return server(); }else if(ac<3) { usage(av[0]); } else if(av[1][0]=='c') { return client(av[2]); } else { usage(av[0]); } return -1; } ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-13 14:50 NFS problems with Linux-2.4 jlnance 2003-05-13 15:19 ` Trond Myklebust @ 2003-05-13 19:07 ` jjs 2003-05-13 19:24 ` Roland Dreier 2003-05-13 23:11 ` Alan Cox 1 sibling, 2 replies; 12+ messages in thread From: jjs @ 2003-05-13 19:07 UTC (permalink / raw) To: jlnance; +Cc: linux kernel jlnance@unity.ncsu.edu wrote: >I have seen the problem on both IA64 machines running the kernel 2.4.18 >from Red Hats Advanced Server > Huh? Advanced server is currently at 2.4.9-e.16 - perhaps this is a self-compiled kernel? Joe ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-13 19:07 ` jjs @ 2003-05-13 19:24 ` Roland Dreier 2003-05-13 21:55 ` jjs 2003-05-13 23:11 ` Alan Cox 1 sibling, 1 reply; 12+ messages in thread From: Roland Dreier @ 2003-05-13 19:24 UTC (permalink / raw) To: jjs; +Cc: jlnance, linux kernel jlnance> I have seen the problem on both IA64 machines running the jlnance> kernel 2.4.18 from Red Hats Advanced Server jjs> Huh? jjs> Advanced server is currently at 2.4.9-e.16 - jjs> perhaps this is a self-compiled kernel? Advanced Server on IA64 is at 2.4.18, NOT 2.4.9. - Roland ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-13 19:24 ` Roland Dreier @ 2003-05-13 21:55 ` jjs 0 siblings, 0 replies; 12+ messages in thread From: jjs @ 2003-05-13 21:55 UTC (permalink / raw) To: Roland Dreier; +Cc: jlnance, linux kernel Roland Dreier wrote: > jlnance> I have seen the problem on both IA64 machines running the > jlnance> kernel 2.4.18 from Red Hats Advanced Server > > jjs> Huh? > jjs> Advanced server is currently at 2.4.9-e.16 - > jjs> perhaps this is a self-compiled kernel? > >Advanced Server on IA64 is at 2.4.18, NOT 2.4.9. > Ah - Thanks for the info, I didn't know about the version skew between platforms... Joe ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NFS problems with Linux-2.4 2003-05-13 19:07 ` jjs 2003-05-13 19:24 ` Roland Dreier @ 2003-05-13 23:11 ` Alan Cox 1 sibling, 0 replies; 12+ messages in thread From: Alan Cox @ 2003-05-13 23:11 UTC (permalink / raw) To: jjs; +Cc: jlnance, Linux Kernel Mailing List On Maw, 2003-05-13 at 20:07, jjs wrote: > jlnance@unity.ncsu.edu wrote: > > >I have seen the problem on both IA64 machines running the kernel 2.4.18 > >from Red Hats Advanced Server > > > Huh? > > Advanced server is currently at 2.4.9-e.16 - > > perhaps this is a self-compiled kernel? AS for IA64 isnt the same kernel as for IA32 ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <482A3FA0050D21419C269D13989C6113127532@lavender-fe.eng.netapp.com>]
* Re: NFS problems with Linux-2.4 [not found] <482A3FA0050D21419C269D13989C6113127532@lavender-fe.eng.netapp.com> @ 2003-05-27 17:29 ` jlnance 0 siblings, 0 replies; 12+ messages in thread From: jlnance @ 2003-05-27 17:29 UTC (permalink / raw) To: linux-kernel; +Cc: SteveD, Charles.Lever Hello All, I wanted to follow up this thread now that I have a working solution. My initial problem was that machine A would create a file and machine B would attempt to stat() or open() it over NFS and it would not be there. I was using the 2.4.7 kernel that came with Red Hat 7.2. Trond suggested I try a more recent kernel since 2.4.7 had known close to open cache consistency problems. I tried the 2.4.20 kernel and it did make the problem better, but it was still there. Someone suggested doing an opendir() to flush the NFS cache. This did make the problem go away with the 2.4.20 kernels. With the 2.4.7 kenrels, I started getting ESTALE errors after I did this. I found that I could work around these errors by doing something like: f = fopen(filename, mode); if(!f) { if(errno==ESTALE) { sleep(1); f = fopen(filename, mode); } } which is ugly, but it allow me to run on unpatched Red Hat 7.2 systems which is highly desirable. Thanks, Jim ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2003-05-27 17:16 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2003-05-13 14:50 NFS problems with Linux-2.4 jlnance 2003-05-13 15:19 ` Trond Myklebust 2003-05-15 15:22 ` Jim Nance 2003-05-18 15:00 ` Trond Myklebust 2003-05-19 0:53 ` jlnance 2003-05-19 11:27 ` Trond Myklebust 2003-05-19 20:02 ` Jim Nance 2003-05-13 19:07 ` jjs 2003-05-13 19:24 ` Roland Dreier 2003-05-13 21:55 ` jjs 2003-05-13 23:11 ` Alan Cox [not found] <482A3FA0050D21419C269D13989C6113127532@lavender-fe.eng.netapp.com> 2003-05-27 17:29 ` jlnance
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).