handling NFSERR_JUKEBOX

handling NFSERR_JUKEBOX

[Bob Smart]

The latest version of unicos has started to return NFSERR_JUKEBOX
when it has to retrieve a file from tape. I found the following
relevant discussion on the netbsd list:
http://mail-index.netbsd.org/tech-kern/1999/03/16/0002.html
To save you looking the key point is:

  >Not sure. Is there a way that the server can say, "I got your request,
  >but I'm too busy now, try again in a little bit." ??

  Isn't this what NFSERR_JUKEBOX is for?

  AFAIK, the protocol goes something like this:

  Client sends a request.
  Server starts loading tape/optical disk/whatever.
  Client resends request.
  Server notices that this is a repeat of an earlier request which is already
  in the "slow queue", and replies NFSERR_JUKEBOX (= "be patient, I'll send
  the response eventually").
  Client shuts up and waits.
  Server completes request and sends response to client.

It seems that what the linux client is doing is returning error 528
to the user program (cp is giving this error message). From 
linux/errno.h:

#define EJUKEBOX  528     /* Request initiated, but will not complete 
                             before timeout */

This is wrong because the nfs file system is hard mounted in my case
- there is no timeout.

While it would be nice to do a perfect solution, it looks like
a quick fix is to just ignore NFSERR_JUKEBOX from the server.

Bob Smart

Re: handling NFSERR_JUKEBOX

[Pete Wyckoff]

smart@hpc.CSIRO.AU said:
> http://mail-index.netbsd.org/tech-kern/1999/03/16/0002.html
> To save you looking the key point is:
> 
>   >Not sure. Is there a way that the server can say, "I got your request,
>   >but I'm too busy now, try again in a little bit." ??
> 
>   Isn't this what NFSERR_JUKEBOX is for?
> 
>   AFAIK, the protocol goes something like this:
> 
>   Client sends a request.
>   Server starts loading tape/optical disk/whatever.
>   Client resends request.
>   Server notices that this is a repeat of an earlier request which is already
>   in the "slow queue", and replies NFSERR_JUKEBOX (= "be patient, I'll send
>   the response eventually").
>   Client shuts up and waits.
>   Server completes request and sends response to client.
> 
> It seems that what the linux client is doing is returning error 528
> to the user program (cp is giving this error message). From 
> linux/errno.h:
> 
> #define EJUKEBOX  528     /* Request initiated, but will not complete 
>                              before timeout */
> 
> This is wrong because the nfs file system is hard mounted in my case
> - there is no timeout.
> 
> While it would be nice to do a perfect solution, it looks like
> a quick fix is to just ignore NFSERR_JUKEBOX from the server.

The comment is incorrect.  Take a look at the RFC for NFSv3.  There is
no duplicate request sent from the client for a JUKEBOX-supporting
server.  The first request is satisfied by the return of EJUKEBOX, after
which it is up to the client to re-request the file.  Ignoring it is not
an option because the request is terminated; a retry does not make
sense.  A new request must be initiated by the client later.

Take a look at the nfs sourceforge mailing list.  There's a long thread
about this in the context of a Solaris HSM server.  I wrote a patch to
try to do the right thing on a linux client, but it is not perfect for
many reasons: http://devrandom.net/lists/archives/2001/6/NFS/0135.html

		-- Pete

Red Hat needs this patch (was Re: handling NFSERR_JUKEBOX)

[Bob Smart]

> Take a look at the nfs sourceforge mailing list.  There's a long thread
> about this in the context of a Solaris HSM server.  I wrote a patch to
> try to do the right thing on a linux client, but it is not perfect for
> many reasons: http://devrandom.net/lists/archives/2001/6/NFS/0135.html

Thanks again for this patch. I've appended our current version to
save others the trouble of minor modifications. I've just tested
it against 2.4.13-ac7. 

Your patch may not be perfect but it is a million times better than
nothing. We have had no problems with it. Linux has a long tradition 
of utilizing imperfect software till something better comes along.

This patch is ESSENTIAL to make Linux useable in large enterprises,
because they increasingly commonly utilize hirearchically managed
storage. I strongly urge Alan/Linus/Macello to include this patch
in the stable kernel tree (but marked experimental). Or at least
can the kernel maintainers at Red Hat and other distributions put
it in their offerings: surely you see that you need this for your
big customers.

I understand that some people think the problem should be addressed 
in glibc (+ all other places that do open system calls). Well really
this is just a small part of a much wider problem. I've included a
rant below. The bottom line is that default "wait as long as necessary"
behaviour should be provided by the kernel, but there needs to be an
optional mechanism that lets the user know when things will take longer
than expected.

Cheers,

Bob

-----------------------------------------------------------------------------

Handling long latency events
----------------------------

The linux kernel potentially returns the error EJUKEBOX from
some operations. This error means: "This operation might well
succeed eventually, and I've started the task [e.g. get
file from tape] that will cause it to later succeed. However
since it will take a long time I'm returning this error so
that you can: inform the user; wait a while; then retry the op."
Now that NFS servers are starting to use NFSERR_JUKEBOX returns,
(which get turned into EJUKEBOX if you don't have the Wyckoff
patch) this is starting to happen and we find that linux user
mode software doesn't handle it.

Getting a file from tape is just one of many circumstances where
operations sometimes take an unexpectedly long time. Other cases
typcially revolve around network operations where remote servers
are not responding for one reason or another. As use of network
services increases this problem is getting worse.

We need to be able to handle these situations in 2 different ways.
If it is a batch or background operation then most of the time
we just want it to wait. Typical example: a telnet server shouldn't
drop a terminal session just because the client has been out of
touch for an hour because of network problems. On the other hand
if the software is interacting with the user and suddenly there
are delays, then we want to give the user useful feedback and
also allow the user to kill the operation. A typical example is
a telnet client [and the (old?) windows behaviour of locking up
is particularly annoying].

It seems to me that the right unixy solution to this problem is
through signals. The EJUKEBOX idea is regretable. However it has
happened and we need to handle it. I strongly urge that:

 - default behaviour is for the kernel to wait and never return 
   EJUKEBOX.

 - however user programs that want to handle EJUKEBOX should
   be able to put themselves into a mode where EJUKEBOX is returned.

This approach has the advantage of minimizing impact on existing
user mode software. If you accept my argument then Pete Wyckoff's
patch is a step in the right direction.

-----------------------------------------------------------------------------

diff -r -u --exclude='*.orig' SAVE/linux-2.4.13-0.3/Documentation/Configure.help linux-2.4.13-0.3+jukebox/Documentation/Configure.help
--- SAVE/linux-2.4.13-0.3/Documentation/Configure.help	Tue Oct 30 09:19:01 2001
+++ linux-2.4.13-0.3+jukebox/Documentation/Configure.help	Sun Nov  4 14:03:59 2001
@@ -14094,6 +14094,23 @@
 
   If unsure, say N.
 
+Auto-retry jukebox error requests (EXPERIMENTAL)
+CONFIG_NFS_V3_JUKEBOX_RETRY
+  Say Y here if you want your NFS client to automatically retry requests
+  to which the server responded with EJUKEBOX (528).  Some NFS servers
+  export filesystems which are hierarchical, in that some files may take
+  much longer than others to read or write.  These files may be migrated
+  to a slow tape storage system which serves as the bulk capacity for the
+  file system; the fast disk becomes essentially a cache.
+
+  If you try to read a file which has been migrated to tape, the server
+  may return the error EJUKEBOX and begin the process which brings the
+  file from tape to disk.  This option enables the kernel to retry
+  the access until the file is ready, rather than passing the error to
+  userspace applications, which is the default.
+ 
+  If unsure, say N.
+
 Root file system on NFS
 CONFIG_ROOT_NFS
   If you want your Linux box to mount its whole root file system (the
diff -r -u --exclude='*.orig' SAVE/linux-2.4.13-0.3/fs/Config.in linux-2.4.13-0.3+jukebox/fs/Config.in
--- SAVE/linux-2.4.13-0.3/fs/Config.in	Tue Oct 30 09:18:42 2001
+++ linux-2.4.13-0.3+jukebox/fs/Config.in	Sun Nov  4 14:03:59 2001
@@ -95,6 +95,7 @@
    dep_tristate 'InterMezzo file system support (experimental, replicating fs)' CONFIG_INTERMEZZO_FS $CONFIG_INET $CONFIG_EXPERIMENTAL
    dep_tristate 'NFS file system support' CONFIG_NFS_FS $CONFIG_INET
    dep_mbool '  Provide NFSv3 client support' CONFIG_NFS_V3 $CONFIG_NFS_FS
+   dep_mbool '    Auto-retry jukebox error requests' CONFIG_NFS_V3_JUKEBOX_RETRY $CONFIG_NFS_V3
    dep_bool '  Root file system on NFS' CONFIG_ROOT_NFS $CONFIG_NFS_FS $CONFIG_IP_PNP
 
    dep_tristate 'NFS server support' CONFIG_NFSD $CONFIG_INET
diff -r -u --exclude='*.orig' SAVE/linux-2.4.13-0.3/fs/nfs/file.c linux-2.4.13-0.3+jukebox/fs/nfs/file.c
--- SAVE/linux-2.4.13-0.3/fs/nfs/file.c	Mon Sep 24 02:48:01 2001
+++ linux-2.4.13-0.3+jukebox/fs/nfs/file.c	Sun Nov  4 15:17:14 2001
@@ -192,8 +192,19 @@
 	if (!inode)
 		return 0;
 
+#ifdef CONFIG_NFS_V3_JUKEBOX_RETRY
+        /* perhaps nfsv3 server returned jukebox, go try again */
+        /* if (PageError(page) && inode->u.nfs_i.async_error == -EJUKEBOX) {
+                result = nfs_readpage_sync(inode->u.nfs_i.file, inode, page); */
+        if (PageError(page) && NFS_I(inode)->async_error == -EJUKEBOX) {
+                result = nfs_readpage_sync(NFS_I(inode)->file, inode, page);
+        } else
+#endif
+        {
 	rpages = NFS_SERVER(inode)->rpages;
 	result = nfs_pagein_inode(inode, index, rpages);
+        }
+
 	if (result < 0)
 		return result;
 	return 0;
diff -r -u --exclude='*.orig' SAVE/linux-2.4.13-0.3/fs/nfs/nfs3proc.c linux-2.4.13-0.3+jukebox/fs/nfs/nfs3proc.c
--- SAVE/linux-2.4.13-0.3/fs/nfs/nfs3proc.c	Tue Oct  2 06:45:37 2001
+++ linux-2.4.13-0.3+jukebox/fs/nfs/nfs3proc.c	Sun Nov  4 16:28:09 2001
@@ -4,6 +4,8 @@
  *  Client-side NFSv3 procedures stubs.
  *
  *  Copyright (C) 1997, Olaf Kirch
+ *
+ *  21 Jun 01 - Retry NFSv3 EJUKEBOX errors by Pete Wyckoff 
  */
 
 #include <linux/mm.h>
@@ -56,10 +58,25 @@
 	struct nfs3_sattrargs	arg = { NFS_FH(inode), sattr, 0, 0 };
 	int	status;
 
+        for (;;) {
 	dprintk("NFS call  setattr\n");
 	fattr->valid = 0;
-	status = rpc_call(NFS_CLIENT(inode), NFS3PROC_SETATTR, &arg, fattr, 0);
-	dprintk("NFS reply setattr\n");
+                status = rpc_call(NFS_CLIENT(inode), NFS3PROC_SETATTR, &arg,
+                    fattr, 0);
+                dprintk("NFS reply setattr %d\n", status);
+#ifdef CONFIG_NFS_V3_JUKEBOX_RETRY
+                if (status == -EJUKEBOX) {
+                        DECLARE_WAIT_QUEUE_HEAD(jukebox_wq);
+                        tty_write_message(current->tty,
+                          "NFSv3: write waiting for file unmigration\r\n");
+                        if (interruptible_sleep_on_timeout(&jukebox_wq,
+                            NFS_JUKEBOX_RETRY_TIMEO) == 0)
+                                continue;  /* but not if interrupted */
+                }
+#endif
+                break;
+        }
+
 	return status;
 }
 
@@ -147,10 +164,24 @@
 	struct rpc_message	msg = { NFS3PROC_READ, &arg, &res, cred };
 	int			status;
 
+        for (;;) {
 	dprintk("NFS call  read %d @ %Ld\n", count, (long long)offset);
 	fattr->valid = 0;
 	status = rpc_call_sync(NFS_CLIENT(inode), &msg, flags);
 	dprintk("NFS reply read: %d\n", status);
+#ifdef CONFIG_NFS_V3_JUKEBOX_RETRY
+                if (status == -EJUKEBOX) {
+                        DECLARE_WAIT_QUEUE_HEAD(jukebox_wq);
+                        tty_write_message(current->tty,
+                          "NFSv3: read waiting for file unmigration\r\n");
+                        if (interruptible_sleep_on_timeout(&jukebox_wq,
+                            NFS_JUKEBOX_RETRY_TIMEO) == 0)
+                                continue;  /* but not if interrupted */
+                }
+#endif
+                break;
+        }
+
 	*eofp = res.eof;
 	return status;
 }
@@ -177,7 +208,7 @@
 
 	status = rpc_call_sync(NFS_CLIENT(inode), &msg, rpcflags);
 
-	dprintk("NFS reply read: %d\n", status);
+	dprintk("NFS reply write: %d\n", status);
 	return status < 0? status : res.count;
 }
 
diff -r -u --exclude='*.orig' SAVE/linux-2.4.13-0.3/fs/nfs/read.c linux-2.4.13-0.3+jukebox/fs/nfs/read.c
--- SAVE/linux-2.4.13-0.3/fs/nfs/read.c	Tue Oct 30 09:18:42 2001
+++ linux-2.4.13-0.3+jukebox/fs/nfs/read.c	Sun Nov  4 15:28:38 2001
@@ -81,7 +81,7 @@
 /*
  * Read a page synchronously.
  */
-static int
+int
 nfs_readpage_sync(struct file *file, struct inode *inode, struct page *page)
 {
 	struct rpc_cred	*cred = NULL;
@@ -449,11 +449,27 @@
 		if (task->tk_status >= 0 && count >= 0) {
 			SetPageUptodate(page);
 			count -= PAGE_CACHE_SIZE;
-		} else
+			flush_dcache_page(page);
+                        kunmap(page);
+                        UnlockPage(page);
+		} else {
 			SetPageError(page);
 		flush_dcache_page(page);
 		kunmap(page);
+#ifdef CONFIG_NFS_V3_JUKEBOX_RETRY
+                        /* do not unlock page, let nfs_sync_page retry */
+                        if (task->tk_status == -EJUKEBOX) {
+                                NFS_I(inode)->async_error = task->tk_status;
+                                if (waitqueue_active(&page->wait))
+                                        wake_up(&page->wait);
+                                else  /* nobody waiting, just unlock */
 		UnlockPage(page);
+                        } else
+#endif
+                        {
+                                UnlockPage(page);
+                        }
+                }
 
 		dprintk("NFS: read (%x/%Ld %d@%Ld)\n",
                         req->wb_inode->i_dev,
@@ -504,8 +520,13 @@
 		goto out_error;
 
 	error = -1;
-	if (!PageError(page) && NFS_SERVER(inode)->rsize >= PAGE_CACHE_SIZE)
+	if (!PageError(page) && NFS_SERVER(inode)->rsize >= PAGE_CACHE_SIZE) {
+#ifdef CONFIG_NFS_V3_JUKEBOX_RETRY
+                NFS_I(inode)->file = file;
+                NFS_I(inode)->async_error = 0;
+#endif
 		error = nfs_readpage_async(file, inode, page);
+	}
 	if (error >= 0)
 		goto out;
 
diff -r -u --exclude='*.orig' SAVE/linux-2.4.13-0.3/include/linux/nfs_fs.h linux-2.4.13-0.3+jukebox/include/linux/nfs_fs.h
--- SAVE/linux-2.4.13-0.3/include/linux/nfs_fs.h	Tue Oct 30 10:04:16 2001
+++ linux-2.4.13-0.3+jukebox/include/linux/nfs_fs.h	Sun Nov  4 14:09:34 2001
@@ -104,6 +104,9 @@
 /* Inode Flags */
 #define NFS_USE_READDIRPLUS(inode)	((NFS_FLAGS(inode) & NFS_INO_ADVISE_RDPLUS) ? 1 : 0)
 
+/* Jukebox retry interval */
+#define NFS_JUKEBOX_RETRY_TIMEO		(10 * HZ)
+
 /*
  * These are the default flags for swap requests
  */
@@ -258,6 +261,7 @@
 extern int  nfs_readpage(struct file *, struct page *);
 extern int  nfs_pagein_inode(struct inode *, unsigned long, unsigned int);
 extern int  nfs_pagein_timeout(struct inode *);
+extern int  nfs_readpage_sync(struct file *, struct inode *, struct page *);
 
 /*
  * linux/fs/mount_clnt.c
diff -r -u --exclude='*.orig' SAVE/linux-2.4.13-0.3/include/linux/nfs_fs_i.h linux-2.4.13-0.3+jukebox/include/linux/nfs_fs_i.h
--- SAVE/linux-2.4.13-0.3/include/linux/nfs_fs_i.h	Tue Oct 30 10:03:46 2001
+++ linux-2.4.13-0.3+jukebox/include/linux/nfs_fs_i.h	Sun Nov  4 14:09:25 2001
@@ -75,6 +75,11 @@
 
 	/* Credentials for shared mmap */
 	struct rpc_cred		*mm_cred;
+
+#ifdef CONFIG_NFS_V3_JUKEBOX_RETRY
+        struct file *file;
+        int async_error;
+#endif
 };
 
 /*

Re: Red Hat needs this patch (was Re: handling NFSERR_JUKEBOX)

[Trond Myklebust]

>>>>> " " == Bob Smart <smart@hpc.CSIRO.AU> writes:

    >> Take a look at the nfs sourceforge mailing list.  There's a
    >> long thread about this in the context of a Solaris HSM server.
    >> I wrote a patch to try to do the right thing on a linux client,
    >> but it is not perfect for many reasons:
    >> http://devrandom.net/lists/archives/2001/6/NFS/0135.html

     > Thanks again for this patch. I've appended our current version
     > to save others the trouble of minor modifications. I've just
     > tested it against 2.4.13-ac7.

     > Your patch may not be perfect but it is a million times better
     > than nothing. We have had no problems with it. Linux has a long
     > tradition of utilizing imperfect software till something better
     > comes along.

     > This patch is ESSENTIAL to make Linux useable in large
     > enterprises, because they increasingly commonly utilize
     > hirearchically managed storage. I strongly urge
     > Alan/Linus/Macello to include this patch in the stable kernel
     > tree (but marked experimental). Or at least can the kernel
     > maintainers at Red Hat and other distributions put it in their
     > offerings: surely you see that you need this for your big
     > customers.

The patch may appear to work, but it fails to understand the nature of
the problem. The problem with NFSERR_JUKEBOX is that is can appear in
more or less *any* NFSv3 RPC call. That includes everything from
lookups to writes.

If all you do is intercept read and setattr, then you certainly
haven't achieved 'enterprise quality'.

Cheers,
   Trond

Re: Red Hat needs this patch (was Re: handling NFSERR_JUKEBOX)

[Marcelo Tosatti]

On 6 Nov 2001, Trond Myklebust wrote:

> >>>>> " " == Bob Smart <smart@hpc.CSIRO.AU> writes:
> 
>     >> Take a look at the nfs sourceforge mailing list.  There's a
>     >> long thread about this in the context of a Solaris HSM server.
>     >> I wrote a patch to try to do the right thing on a linux client,
>     >> but it is not perfect for many reasons:
>     >> http://devrandom.net/lists/archives/2001/6/NFS/0135.html
> 
>      > Thanks again for this patch. I've appended our current version
>      > to save others the trouble of minor modifications. I've just
>      > tested it against 2.4.13-ac7.
> 
>      > Your patch may not be perfect but it is a million times better
>      > than nothing. We have had no problems with it. Linux has a long
>      > tradition of utilizing imperfect software till something better
>      > comes along.
> 
>      > This patch is ESSENTIAL to make Linux useable in large
>      > enterprises, because they increasingly commonly utilize
>      > hirearchically managed storage. I strongly urge
>      > Alan/Linus/Macello to include this patch in the stable kernel
>      > tree (but marked experimental). Or at least can the kernel
>      > maintainers at Red Hat and other distributions put it in their
>      > offerings: surely you see that you need this for your big
>      > customers.
> 
> The patch may appear to work, but it fails to understand the nature of
> the problem. The problem with NFSERR_JUKEBOX is that is can appear in
> more or less *any* NFSv3 RPC call. That includes everything from
> lookups to writes.
> 
> If all you do is intercept read and setattr, then you certainly
> haven't achieved 'enterprise quality'.

Trond, 

Should'nt we make Linux retry on NFSERR_JUKEBOX _by default_ ? 

Is there any reason why we already don't do that already except "nobody
coded it yet" ? 

Re: Red Hat needs this patch (was Re: handling NFSERR_JUKEBOX)

[Bob Smart]

> If all you do is intercept read and setattr, then you certainly
> haven't achieved 'enterprise quality'.

I don't have documentation on when Cray's NFSv3 server will return
NFSERR_JUKEBOX, but since inodes aren't migrated to tape I don't
think it is likely on other than read and non-truncate write with
no prior read. Our applications are unlikely to do the latter.

Maybe there is a subset of servers for which the patch is OK, and
we could make that clear in the help for the config option. I'll
ask Cray for clarification about their server.

Bob

Re: Red Hat needs this patch (was Re: handling NFSERR_JUKEBOX)

[Trond Myklebust]

>>>>> " " == Marcelo Tosatti <marcelo@conectiva.com.br> writes:

     > Trond,

     > Should'nt we make Linux retry on NFSERR_JUKEBOX _by default_ ?

Normally we probably should. The thing that worries me is the fact
that we may be holding the VFS semaphores for long periods of
time. In particular that will greatly slow down lookups...
This is the reason why I'd prefer a lot of these things to be done in
the VFS layer or above.

Cheers,
  Trond

Re: Red Hat needs this patch (was Re: handling NFSERR_JUKEBOX)

[Trond Myklebust]

>>>>> " " == Bob Smart <smart@hpc.CSIRO.AU> writes:

     > Maybe there is a subset of servers for which the patch is OK,
     > and we could make that clear in the help for the config
     > option. I'll ask Cray for clarification about their server.

I'm still not convinced this is a good idea, but if you are going to
do things inside the NFS client, why don't you instead write a wrapper
function around rpc_call_sync() for fs/nfs/nfs3proc.c. Something like

static int
nfs_rpc_wrapper(struct rpc_clnt *clnt, struct rpc_message *msg, int flags)
{
        sigset_t oldset;
        int res;
        rpc_clnt_sigmask(clnt, &oldset);
        do {
                res = rpc_call_sync(clnt, msg, flags);
                if (result != -EJUKEBOX)
                        break;
                set_current_state(TASK_INTERRUPTIBLE);
                schedule_timeout(NFS_JUKEBOX_RETRY_TIME);
                res = -ERESTARTSYS;
        } while (!signalled());
        rpc_clnt_sigunmask(clnt, &oldset);
        return res;
}

and then use a couple of '#define's to wrap rpc_call() and
rpc_call_sync() in nfs3_proc_*(). That will take care of all those
synchronous calls in one fell swoop.

You'll still need to take care of asynchronous reads, writes and
unlink, but those are easy. Just do something like

if (task->tk_error == -EJUKEBOX) {
        rpc_delay(task, NFS_JUKEBOX_RETRY_TIMEO);
        rpc_restart_call(task);
        return;
}

in nfs_readpage_result(), nfs_writeback_done(), nfs_commit_done(), and
nfs_async_unlink_done()...

Cheers,
   Trond

Re: handling NFSERR_JUKEBOX

[Trond Myklebust]

>>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:

     > I'm still not convinced this is a good idea, but if you are
     > going to do things inside the NFS client, why don't you instead
     > write a wrapper function around rpc_call_sync() for
     > fs/nfs/nfs3proc.c. Something like

...and here's the full patch that implements it...

Cheers,
  Trond

diff -u --recursive --new-file linux-2.4.14/fs/nfs/nfs3proc.c linux-2.4.14-jukebox/fs/nfs/nfs3proc.c
--- linux-2.4.14/fs/nfs/nfs3proc.c	Mon Oct  1 22:45:37 2001
+++ linux-2.4.14-jukebox/fs/nfs/nfs3proc.c	Wed Nov  7 00:25:19 2001
@@ -17,6 +17,37 @@
 
 #define NFSDBG_FACILITY		NFSDBG_PROC
 
+/* A wrapper to handle the EJUKEBOX error message */
+static int
+nfs3_rpc_wrapper(struct rpc_clnt *clnt, struct rpc_message *msg, int flags)
+{
+	sigset_t oldset;
+	int res;
+	rpc_clnt_sigmask(clnt, &oldset);
+	do {
+		res = rpc_call_sync(clnt, msg, flags);
+		if (res != -EJUKEBOX)
+			break;
+		set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(NFS_JUKEBOX_RETRY_TIME);
+		res = -ERESTARTSYS;
+	} while (!signalled());
+	rpc_clnt_sigunmask(clnt, &oldset);
+	return res;
+}
+
+static inline int
+nfs3_rpc_call_wrapper(struct rpc_clnt *clnt, u32 proc, void *argp, void *resp, int flags)
+{
+	struct rpc_message msg = { proc, argp, resp, NULL };
+	return nfs3_rpc_wrapper(clnt, &msg, flags);
+}
+
+#define rpc_call(clnt, proc, argp, resp, flags) \
+		nfs3_rpc_call_wrapper(clnt, proc, argp, resp, flags)
+#define rpc_call_sync(clnt, msg, flags) \
+		nfs3_rpc_wrapper(clnt, msg, flags)
+
 /*
  * Bare-bones access to getattr: this is for nfs_read_super.
  */
diff -u --recursive --new-file linux-2.4.14/fs/nfs/read.c linux-2.4.14-jukebox/fs/nfs/read.c
--- linux-2.4.14/fs/nfs/read.c	Thu Oct 11 17:12:52 2001
+++ linux-2.4.14-jukebox/fs/nfs/read.c	Wed Nov  7 00:25:19 2001
@@ -437,6 +437,9 @@
 	dprintk("NFS: %4d nfs_readpage_result, (status %d)\n",
 		task->tk_pid, task->tk_status);
 
+	if (nfs_async_handle_jukebox(task))
+		return;
+
 	nfs_refresh_inode(inode, &data->fattr);
 	while (!list_empty(&data->pages)) {
 		struct nfs_page *req = nfs_list_entry(data->pages.next);
diff -u --recursive --new-file linux-2.4.14/fs/nfs/unlink.c linux-2.4.14-jukebox/fs/nfs/unlink.c
--- linux-2.4.14/fs/nfs/unlink.c	Thu Aug 16 18:39:37 2001
+++ linux-2.4.14-jukebox/fs/nfs/unlink.c	Wed Nov  7 00:25:19 2001
@@ -123,6 +123,8 @@
 	struct dentry		*dir = data->dir;
 	struct inode		*dir_i;
 
+	if (nfs_async_handle_jukebox(task))
+		return;
 	if (!dir)
 		return;
 	dir_i = dir->d_inode;
diff -u --recursive --new-file linux-2.4.14/fs/nfs/write.c linux-2.4.14-jukebox/fs/nfs/write.c
--- linux-2.4.14/fs/nfs/write.c	Thu Oct 11 17:12:52 2001
+++ linux-2.4.14-jukebox/fs/nfs/write.c	Wed Nov  7 00:25:19 2001
@@ -1229,6 +1229,9 @@
 	dprintk("NFS: %4d nfs_writeback_done (status %d)\n",
 		task->tk_pid, task->tk_status);
 
+	if (nfs_async_handle_jukebox(task))
+		return;
+
 	/* We can't handle that yet but we check for it nevertheless */
 	if (resp->count < argp->count && task->tk_status >= 0) {
 		static unsigned long    complain;
@@ -1422,6 +1425,9 @@
         dprintk("NFS: %4d nfs_commit_done (status %d)\n",
                                 task->tk_pid, task->tk_status);
 
+	if (nfs_async_handle_jukebox(task))
+		return;
+
 	nfs_write_attributes(inode, resp->fattr);
 	while (!list_empty(&data->pages)) {
 		req = nfs_list_entry(data->pages.next);
diff -u --recursive --new-file linux-2.4.14/include/linux/nfs_fs.h linux-2.4.14-jukebox/include/linux/nfs_fs.h
--- linux-2.4.14/include/linux/nfs_fs.h	Wed Oct 24 07:00:07 2001
+++ linux-2.4.14-jukebox/include/linux/nfs_fs.h	Wed Nov  7 00:25:19 2001
@@ -16,6 +16,7 @@
 
 #include <linux/sunrpc/debug.h>
 #include <linux/sunrpc/auth.h>
+#include <linux/sunrpc/clnt.h>
 
 #include <linux/nfs.h>
 #include <linux/nfs2.h>
@@ -326,6 +327,29 @@
 	__retval;							\
 })
 
+#ifdef CONFIG_NFS_V3
+
+#define NFS_JUKEBOX_RETRY_TIME (5 * HZ)
+static inline int
+nfs_async_handle_jukebox(struct rpc_task *task)
+{
+	if (task->tk_status != -EJUKEBOX)
+		return 0;
+	task->tk_status = 0;
+	rpc_restart_call(task);
+	rpc_delay(task, NFS_JUKEBOX_RETRY_TIME);
+	return 1;
+}
+
+#else
+
+static inline int
+nfs_async_handle_jukebox(struct rpc_task *task)
+{
+	return 0;
+}
+#endif /* CONFIG_NFS_V3 */
+
 #endif /* __KERNEL__ */
 
 /*

Re: handling NFSERR_JUKEBOX

[Pete Wyckoff]

trond.myklebust@fys.uio.no said:
> > I'm still not convinced this is a good idea, but if you are
> > going to do things inside the NFS client, why don't you instead
> > write a wrapper function around rpc_call_sync() for
> > fs/nfs/nfs3proc.c. Something like
> 
> ...and here's the full patch that implements it...

Works like a charm.  Glad you allowed yourself to be badgered
into writing the proper code to handle EJUKEBOX.  I'll destroy
my icky version.

Can you push it forward to Linus for general inclusion?

		-- Pete