[PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[andros]

Responded to comments, added a 2 cleanup patchses

Plus some code cleanup
0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch

and some bug fixes
0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch

NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch
was replaced by:
0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch


Remove unused (by file layout) encode_layoutreturn io operation
0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch

0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch

Note: pnfs4_proc_layoutget is only called by send_layout() which prints
the status.
0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch

Add back encode_layoutreturn io operation
0008-pnfs-post-submit-restore-encode_layoutreturn.patch


New patches:
0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch

This gets rid of a frame stack warning;
0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch

Testing:
---------

CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
Passes Connectathon tests

Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the
pyNFS server and the testclient framework.

Still todo:

Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and
layoutcommit do not pass nfs_stste to the error handlers.

Handle NFS4ERR_BAD_LAYOUT.

CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.

-->Andy

[PATCH 01/10] SQUASHME: pnfs-submit remove unused filelayout_mount_type

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.h |    6 ------
 1 files changed, 0 insertions(+), 6 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index 29e481d..de8391f 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -24,8 +24,6 @@
 #define NFS4_PNFS_MAX_MULTI_CNT  64 /* 256 fit into a u8 stripe_index */
 #define NFS4_PNFS_MAX_MULTI_DS   2
 
-#define FILE_MT(inode) ((struct filelayout_mount_type *) \
-			(NFS_SERVER(inode)->pnfs_mountid->mountid))
 #define FILE_DSADDR(lseg) (container_of(lseg->deviceid, \
 					struct nfs4_file_layout_dsaddr, \
 					deviceid))
@@ -76,10 +74,6 @@ struct nfs4_filelayout {
 	u32 stripe_unit;
 };
 
-struct filelayout_mount_type {
-	struct super_block *fl_sb;
-};
-
 extern struct nfs_fh *
 nfs4_fl_select_ds_fh(struct pnfs_layout_segment *lseg, loff_t offset);
 
-- 
1.6.6

[PATCH 02/10] SQUASHME pnfs-submit: pnfs_try_to_read, write, commit using freed memory

[andros]

From: Andy Adamson <andros@netapp.com>

The nfs_read_data or nfs_write_data is freed during rpc call.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/pnfs.h |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index 78776f3..c60eff6 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -105,13 +105,14 @@ static inline enum pnfs_try_status
 pnfs_try_to_read_data(struct nfs_read_data *data,
 		      const struct rpc_call_ops *call_ops)
 {
+	struct inode *inode = data->inode;
 	enum pnfs_try_status ret;
 
 	if (!data->req->wb_lseg)
 		return PNFS_NOT_ATTEMPTED;
 	ret = _pnfs_try_to_read_data(data, call_ops);
 	if (ret == PNFS_ATTEMPTED)
-		nfs_inc_stats(data->inode, NFSIOS_PNFS_READ);
+		nfs_inc_stats(inode, NFSIOS_PNFS_READ);
 	else
 		_pnfs_clear_lseg_from_pages(&data->pages);
 	return ret;
@@ -122,13 +123,14 @@ pnfs_try_to_write_data(struct nfs_write_data *data,
 		       const struct rpc_call_ops *call_ops,
 		       int how)
 {
+	struct inode *inode = data->inode;
 	enum pnfs_try_status ret;
 
 	if (!data->req->wb_lseg)
 		return PNFS_NOT_ATTEMPTED;
 	ret = _pnfs_try_to_write_data(data, call_ops, how);
 	if (ret == PNFS_ATTEMPTED)
-		nfs_inc_stats(data->inode, NFSIOS_PNFS_WRITE);
+		nfs_inc_stats(inode, NFSIOS_PNFS_WRITE);
 	else
 		_pnfs_clear_lseg_from_pages(&data->pages);
 	return ret;
@@ -139,6 +141,7 @@ pnfs_try_to_commit(struct nfs_write_data *data,
 		   const struct rpc_call_ops *call_ops,
 		   int how)
 {
+	struct inode *inode = data->inode;
 	enum pnfs_try_status ret;
 
 	/* Unlike in pnfs_try_to_write_data and pnfs_try_to_read_data,
@@ -148,7 +151,7 @@ pnfs_try_to_commit(struct nfs_write_data *data,
 	 */
 	ret = _pnfs_try_to_commit(data, call_ops, how);
 	if (ret == PNFS_ATTEMPTED)
-		nfs_inc_stats(data->inode, NFSIOS_PNFS_COMMIT);
+		nfs_inc_stats(inode, NFSIOS_PNFS_COMMIT);
 	else
 		_pnfs_clear_lseg_from_pages(&data->pages);
 	return ret;
-- 
1.6.6

[PATCH 03/10] SQUASHME pnfs-submit: clear page lseg on partial i/o

[andros]

From: Andy Adamson <andros@netapp.com>

partial reads or writes use a single page not the page list.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/pnfs.h |   16 ++++++++++++----
 1 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index c60eff6..70f1d97 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -91,10 +91,18 @@ static inline int pnfs_enabled_sb(struct nfs_server *nfss)
 	return nfss->pnfs_curr_ld != NULL;
 }
 
-static inline void _pnfs_clear_lseg_from_pages(struct list_head *head)
+static inline void _pnfs_clear_lseg_from_pages(struct nfs_page *partial,
+					       struct list_head *head)
 {
 	struct nfs_page *req;
 
+	/* partial_read or partial_write */
+	if (list_empty(head)) {
+		put_lseg(partial->wb_lseg);
+		partial->wb_lseg = NULL;
+		return;
+	}
+	/* full_read or full_write */
 	list_for_each_entry(req, head, wb_list) {
 		put_lseg(req->wb_lseg);
 		req->wb_lseg = NULL;
@@ -114,7 +122,7 @@ pnfs_try_to_read_data(struct nfs_read_data *data,
 	if (ret == PNFS_ATTEMPTED)
 		nfs_inc_stats(inode, NFSIOS_PNFS_READ);
 	else
-		_pnfs_clear_lseg_from_pages(&data->pages);
+		_pnfs_clear_lseg_from_pages(data->req, &data->pages);
 	return ret;
 }
 
@@ -132,7 +140,7 @@ pnfs_try_to_write_data(struct nfs_write_data *data,
 	if (ret == PNFS_ATTEMPTED)
 		nfs_inc_stats(inode, NFSIOS_PNFS_WRITE);
 	else
-		_pnfs_clear_lseg_from_pages(&data->pages);
+		_pnfs_clear_lseg_from_pages(data->req, &data->pages);
 	return ret;
 }
 
@@ -153,7 +161,7 @@ pnfs_try_to_commit(struct nfs_write_data *data,
 	if (ret == PNFS_ATTEMPTED)
 		nfs_inc_stats(inode, NFSIOS_PNFS_COMMIT);
 	else
-		_pnfs_clear_lseg_from_pages(&data->pages);
+		_pnfs_clear_lseg_from_pages(data->req, &data->pages);
 	return ret;
 }
 
-- 
1.6.6

[PATCH 04/10] SQUASHME pnfs-submit: remove encode_layoutreturn

[andros]

From: Andy Adamson <andros@netapp.com>

Not used by the file layout

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4xdr.c          |   15 ++-------------
 include/linux/nfs4_pnfs.h |    4 ----
 2 files changed, 2 insertions(+), 17 deletions(-)

diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index a509bc1..fb50361 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -1874,24 +1874,13 @@ encode_layoutreturn(struct xdr_stream *xdr,
 	*p++ = cpu_to_be32(args->lseg.iomode);
 	*p = cpu_to_be32(args->return_type);
 	if (args->return_type == RETURN_FILE) {
-		struct layoutdriver_io_operations *ld_io_ops =
-			NFS_SERVER(args->inode)->pnfs_curr_ld->ld_io_ops;
-
 		p = reserve_space(xdr, 16 + NFS4_STATEID_SIZE);
 		p = xdr_encode_hyper(p, args->lseg.offset);
 		p = xdr_encode_hyper(p, args->lseg.length);
 		p = xdr_encode_opaque_fixed(p, &args->stateid.u.data,
 					    NFS4_STATEID_SIZE);
-
-		dprintk("%s: call %pF\n", __func__,
-			ld_io_ops->encode_layoutreturn);
-		if (ld_io_ops->encode_layoutreturn) {
-			ld_io_ops->encode_layoutreturn(
-				&NFS_I(args->inode)->layout, xdr, args);
-		} else {
-			p = reserve_space(xdr, 4);
-			*p = cpu_to_be32(0);
-		}
+		p = reserve_space(xdr, 4);
+		*p = cpu_to_be32(0);
 	}
 	hdr->nops++;
 	hdr->replen += decode_layoutreturn_maxsz;
diff --git a/include/linux/nfs4_pnfs.h b/include/linux/nfs4_pnfs.h
index d20b5de..782fdd9 100644
--- a/include/linux/nfs4_pnfs.h
+++ b/include/linux/nfs4_pnfs.h
@@ -154,10 +154,6 @@ struct layoutdriver_io_operations {
 	struct pnfs_layout_segment * (*alloc_lseg) (struct pnfs_layout_type *layoutid, struct nfs4_pnfs_layoutget_res *lgr);
 	void (*free_lseg) (struct pnfs_layout_segment *lseg);
 
-	void (*encode_layoutreturn) (struct pnfs_layout_type *layoutid,
-				struct xdr_stream *xdr,
-				const struct nfs4_pnfs_layoutreturn_arg *args);
-
 	/* Registration information for a new mounted file system
 	 */
 	int (*initialize_mountpoint) (struct nfs_client *);
-- 
1.6.6

[PATCH 05/10] SQUASHME pnfs-submit: add error handling to layout return

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4proc.c |   37 +++++++++++++++++++++++++++----------
 1 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 5ffdfe1..0ab875f 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5641,6 +5641,11 @@ static void nfs4_pnfs_layoutreturn_done(struct rpc_task *task, void *calldata)
 	dprintk("--> %s\n", __func__);
 
 	nfs4_sequence_done(server, &lrp->res.seq_res, task->tk_status);
+	if (RPC_ASSASSINATED(task))
+		return;
+
+	if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN)
+		nfs_restart_rpc(task, server->nfs_client);
 
 	dprintk("<-- %s\n", __func__);
 }
@@ -5667,7 +5672,7 @@ static const struct rpc_call_ops nfs4_pnfs_layoutreturn_call_ops = {
 	.rpc_release = nfs4_pnfs_layoutreturn_release,
 };
 
-int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool wait)
+int _pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool issync)
 {
 	struct inode *ino = lrp->args.inode;
 	struct nfs_server *server = NFS_SERVER(ino);
@@ -5689,21 +5694,33 @@ int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool wait)
 	dprintk("--> %s\n", __func__);
 	lrp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE;
 	task = rpc_run_task(&task_setup_data);
-	if (IS_ERR(task)) {
-		status = PTR_ERR(task);
+	if (IS_ERR(task))
+		return PTR_ERR(task);
+	if (!issync)
 		goto out;
-	}
-	if (wait) {
-		status = nfs4_wait_for_completion_rpc_task(task);
-		if (status == 0)
-			status = task->tk_status;
-	}
-	rpc_put_task(task);
+	status = nfs4_wait_for_completion_rpc_task(task);
+	if (status != 0)
+		goto out;
+	status = task->tk_status;
 out:
 	dprintk("<-- %s\n", __func__);
+	rpc_put_task(task);
 	return status;
 }
 
+int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool issync)
+{
+	struct nfs_server *server = NFS_SERVER(lrp->args.inode);
+	struct nfs4_exception exception = { };
+	int err;
+	do {
+		err = nfs4_handle_exception(server,
+				_pnfs4_proc_layoutreturn(lrp, issync),
+				&exception);
+	} while (exception.retry);
+	return err;
+}
+
 int nfs4_pnfs_getdeviceinfo(struct nfs_server *server, struct pnfs_device *pdev)
 {
 	struct nfs4_pnfs_getdeviceinfo_arg args = {
-- 
1.6.6

[PATCH 06/10] SQUASHME pnfs-submit: handle assassinated layoutcommit

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4proc.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 0ab875f..35e8a56 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5539,6 +5539,8 @@ pnfs_layoutcommit_done(struct rpc_task *task, void *calldata)
 	data->status = task->tk_status;
 
 	nfs4_sequence_done(server, &data->res.seq_res, task->tk_status);
+	if (RPC_ASSASSINATED(task))
+		return;
 
 	if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN)
 		nfs_restart_rpc(task, server->nfs_client);
-- 
1.6.6

[PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get

[andros]

From: Andy Adamson <andros@netapp.com>

Redo pnfs4_proc_layoutget() logic to match the other async operations that call
nfs4_wait_for_completion_rpc_task.

pnfs_get_layout_done passes session and delay errors. Handle them with the
generic error handlers.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4proc.c |   45 +++++++++++++++++++++++++++++----------------
 1 files changed, 29 insertions(+), 16 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 35e8a56..6283996 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5446,6 +5446,10 @@ static void nfs4_pnfs_layoutget_done(struct rpc_task *task, void *calldata)
 		return;
 
 	pnfs_get_layout_done(lgp, task->tk_status);
+
+	if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN)
+		nfs_restart_rpc(task, server->nfs_client);
+
 	dprintk("<-- %s\n", __func__);
 }
 
@@ -5471,10 +5475,9 @@ static const struct rpc_call_ops nfs4_pnfs_layoutget_call_ops = {
  * and deal with retries.
  * Currently we can't since we release lgp and its contents.
  */
-int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
+static int _pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
 {
-	struct inode *ino = lgp->args.inode;
-	struct nfs_server *server = NFS_SERVER(ino);
+	struct nfs_server *server = NFS_SERVER(lgp->args.inode);
 	struct rpc_task *task;
 	struct rpc_message msg = {
 		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_PNFS_LAYOUTGET],
@@ -5488,35 +5491,45 @@ int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
 		.callback_data = lgp,
 		.flags = RPC_TASK_ASYNC,
 	};
-	int status;
+	int status = 0;
 
 	dprintk("--> %s\n", __func__);
 
 	lgp->res.layout.buf = (void *)__get_free_page(GFP_NOFS);
 	if (lgp->res.layout.buf == NULL) {
 		nfs4_pnfs_layoutget_release(lgp);
-		status = -ENOMEM;
-		goto out;
+		return -ENOMEM;
 	}
 
 	lgp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE;
 	task = rpc_run_task(&task_setup_data);
-	if (IS_ERR(task)) {
-		status = PTR_ERR(task);
-		goto out;
-	}
+	if (IS_ERR(task))
+		return PTR_ERR(task);
 	status = nfs4_wait_for_completion_rpc_task(task);
-	if (status == 0) {
-		status = lgp->status;
-		if (status == 0)
-			status = pnfs_layout_process(lgp);
-	}
-	rpc_put_task(task);
+	if (status != 0)
+		goto out;
+	status = lgp->status;
+	if (status != 0)
+		goto out;
+	status = pnfs_layout_process(lgp);
 out:
+	rpc_put_task(task);
 	dprintk("<-- %s status=%d\n", __func__, status);
 	return status;
 }
 
+int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
+{
+	struct nfs_server *server = NFS_SERVER(lgp->args.inode);
+	struct nfs4_exception exception = { };
+	int err;
+	do {
+		err = nfs4_handle_exception(server, _pnfs4_proc_layoutget(lgp),
+					    &exception);
+	} while (exception.retry);
+	return err;
+}
+
 static void pnfs_layoutcommit_prepare(struct rpc_task *task, void *data)
 {
 	struct pnfs_layoutcommit_data *ldata =
-- 
1.6.6

[PATCH 08/10] pnfs-post-submit: restore encode_layoutreturn

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4xdr.c          |   14 ++++++++++++--
 include/linux/nfs4_pnfs.h |    4 ++++
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index fb50361..eeee855 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -1874,13 +1874,23 @@ encode_layoutreturn(struct xdr_stream *xdr,
 	*p++ = cpu_to_be32(args->lseg.iomode);
 	*p = cpu_to_be32(args->return_type);
 	if (args->return_type == RETURN_FILE) {
+		struct layoutdriver_io_operations *ld_io_ops =
+			NFS_SERVER(args->inode)->pnfs_curr_ld->ld_io_ops;
+
 		p = reserve_space(xdr, 16 + NFS4_STATEID_SIZE);
 		p = xdr_encode_hyper(p, args->lseg.offset);
 		p = xdr_encode_hyper(p, args->lseg.length);
 		p = xdr_encode_opaque_fixed(p, &args->stateid.u.data,
 					    NFS4_STATEID_SIZE);
-		p = reserve_space(xdr, 4);
-		*p = cpu_to_be32(0);
+		dprintk("%s: call %pF\n", __func__,
+		ld_io_ops->encode_layoutreturn);
+		if (ld_io_ops->encode_layoutreturn) {
+			ld_io_ops->encode_layoutreturn(
+				&NFS_I(args->inode)->layout, xdr, args);
+		} else {
+			p = reserve_space(xdr, 4);
+			*p = cpu_to_be32(0);
+		}
 	}
 	hdr->nops++;
 	hdr->replen += decode_layoutreturn_maxsz;
diff --git a/include/linux/nfs4_pnfs.h b/include/linux/nfs4_pnfs.h
index 782fdd9..d20b5de 100644
--- a/include/linux/nfs4_pnfs.h
+++ b/include/linux/nfs4_pnfs.h
@@ -154,6 +154,10 @@ struct layoutdriver_io_operations {
 	struct pnfs_layout_segment * (*alloc_lseg) (struct pnfs_layout_type *layoutid, struct nfs4_pnfs_layoutget_res *lgr);
 	void (*free_lseg) (struct pnfs_layout_segment *lseg);
 
+	void (*encode_layoutreturn) (struct pnfs_layout_type *layoutid,
+				struct xdr_stream *xdr,
+				const struct nfs4_pnfs_layoutreturn_arg *args);
+
 	/* Registration information for a new mounted file system
 	 */
 	int (*initialize_mountpoint) (struct nfs_client *);
-- 
1.6.6

[PATCH 09/10] SQUASHME: pnfs-submit: don't re-initialize i_lock

[andros]

From: Andy Adamson <andros@netapp.com>

Initialized in the VFS (inode_init_always)

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/inode.c |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)

diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index d6dd6e5..7989dea 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -1420,7 +1420,6 @@ static void pnfs_init_once(struct nfs_inode *nfsi)
 {
 #ifdef CONFIG_NFS_V4_1
 	init_waitqueue_head(&nfsi->lo_waitq);
-	spin_lock_init(&nfsi->vfs_inode.i_lock);
 	seqlock_init(&nfsi->layout.seqlock);
 	INIT_LIST_HEAD(&nfsi->layout.lo_layouts);
 	INIT_LIST_HEAD(&nfsi->layout.segs);
-- 
1.6.6

[PATCH 10/10] SQUASHME pnfs-submit: remove struct nfs_server from stack

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |   18 ++++++++++++------
 1 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 5842c09..8e9d04a 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -121,9 +121,7 @@ _data_server_lookup(u32 ip_addr, u32 port)
 static int
 nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds)
 {
-	struct nfs_server	tmp = {
-		.nfs_client = NULL,
-	};
+	struct nfs_server	*tmp;
 	struct sockaddr_in	sin;
 	struct rpc_clnt 	*mds_clnt = mds_srv->client;
 	struct nfs_client	*clp = mds_srv->nfs_client;
@@ -154,12 +152,18 @@ nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds)
 		}
 		goto out;
 	}
+
+	/* Temporay server for nfs4_set_client */
+	tmp = kzalloc(sizeof(struct nfs_server), GFP_KERNEL);
+	if (!tmp)
+		goto out;
+
 	/*
 	 * Set a retrans, timeout interval, and authflavor equual to the MDS
 	 * values. Use the MDS nfs_client cl_ipaddr field so as to use the
 	 * same co_ownerid as the MDS.
 	 */
-	err = nfs4_set_client(&tmp,
+	err = nfs4_set_client(tmp,
 			      mds_srv->nfs_client->cl_hostname,
 			      (struct sockaddr *)&sin,
 			      sizeof(struct sockaddr),
@@ -169,9 +173,9 @@ nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds)
 			      mds_clnt->cl_xprt->timeout,
 			      1 /* minorversion */);
 	if (err < 0)
-		goto out;
+		goto out_free;
 
-	clp = tmp.nfs_client;
+	clp = tmp->nfs_client;
 
 	/* Ask for only the EXCHGID4_FLAG_USE_PNFS_DS pNFS role */
 	dprintk("%s EXCHANGE_ID for clp %p\n", __func__, clp);
@@ -211,6 +215,8 @@ nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds)
 	dprintk("%s: ip=%x, port=%hu, rpcclient %p\n", __func__,
 				ntohl(ds->ds_ip_addr), ntohs(ds->ds_port),
 				clp->cl_rpcclient);
+out_free:
+	kfree(tmp);
 out:
 	dprintk("%s Returns %d\n", __func__, err);
 	return err;
-- 
1.6.6

Re: [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get

[Boaz Harrosh]

On 06/23/2010 10:21 PM, andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Redo pnfs4_proc_layoutget() logic to match the other async operations that call
> nfs4_wait_for_completion_rpc_task.
> 
> pnfs_get_layout_done passes session and delay errors. Handle them with the
> generic error handlers.
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/nfs4proc.c |   45 +++++++++++++++++++++++++++++----------------
>  1 files changed, 29 insertions(+), 16 deletions(-)
> 
> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
> index 35e8a56..6283996 100644
> --- a/fs/nfs/nfs4proc.c
> +++ b/fs/nfs/nfs4proc.c
> @@ -5446,6 +5446,10 @@ static void nfs4_pnfs_layoutget_done(struct rpc_task *task, void *calldata)
>  		return;
>  
>  	pnfs_get_layout_done(lgp, task->tk_status);
> +
> +	if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN)
> +		nfs_restart_rpc(task, server->nfs_client);
> +

Andy high.

In pnfs_get_layout_done() it is working hard to do "lgp->status = XXX"

I never understood, what gets to look at that later on?

Thanks
Boaz

>  	dprintk("<-- %s\n", __func__);
>  }
>  
> @@ -5471,10 +5475,9 @@ static const struct rpc_call_ops nfs4_pnfs_layoutget_call_ops = {
>   * and deal with retries.
>   * Currently we can't since we release lgp and its contents.
>   */
> -int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
> +static int _pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
>  {
> -	struct inode *ino = lgp->args.inode;
> -	struct nfs_server *server = NFS_SERVER(ino);
> +	struct nfs_server *server = NFS_SERVER(lgp->args.inode);
>  	struct rpc_task *task;
>  	struct rpc_message msg = {
>  		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_PNFS_LAYOUTGET],
> @@ -5488,35 +5491,45 @@ int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
>  		.callback_data = lgp,
>  		.flags = RPC_TASK_ASYNC,
>  	};
> -	int status;
> +	int status = 0;
>  
>  	dprintk("--> %s\n", __func__);
>  
>  	lgp->res.layout.buf = (void *)__get_free_page(GFP_NOFS);
>  	if (lgp->res.layout.buf == NULL) {
>  		nfs4_pnfs_layoutget_release(lgp);
> -		status = -ENOMEM;
> -		goto out;
> +		return -ENOMEM;
>  	}
>  
>  	lgp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE;
>  	task = rpc_run_task(&task_setup_data);
> -	if (IS_ERR(task)) {
> -		status = PTR_ERR(task);
> -		goto out;
> -	}
> +	if (IS_ERR(task))
> +		return PTR_ERR(task);
>  	status = nfs4_wait_for_completion_rpc_task(task);
> -	if (status == 0) {
> -		status = lgp->status;
> -		if (status == 0)
> -			status = pnfs_layout_process(lgp);
> -	}
> -	rpc_put_task(task);
> +	if (status != 0)
> +		goto out;
> +	status = lgp->status;
> +	if (status != 0)
> +		goto out;
> +	status = pnfs_layout_process(lgp);
>  out:
> +	rpc_put_task(task);
>  	dprintk("<-- %s status=%d\n", __func__, status);
>  	return status;
>  }
>  
> +int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
> +{
> +	struct nfs_server *server = NFS_SERVER(lgp->args.inode);
> +	struct nfs4_exception exception = { };
> +	int err;
> +	do {
> +		err = nfs4_handle_exception(server, _pnfs4_proc_layoutget(lgp),
> +					    &exception);
> +	} while (exception.retry);
> +	return err;
> +}
> +
>  static void pnfs_layoutcommit_prepare(struct rpc_task *task, void *data)
>  {
>  	struct pnfs_layoutcommit_data *ldata =

Re: [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get

[William A. (Andy) Adamson]

On Wed, Jun 30, 2010 at 11:19 AM, Boaz Harrosh <bharrosh@panasas.com> w=
rote:
> On 06/23/2010 10:21 PM, andros@netapp.com wrote:
>> From: Andy Adamson <andros@netapp.com>
>>
>> Redo pnfs4_proc_layoutget() logic to match the other async operation=
s that call
>> nfs4_wait_for_completion_rpc_task.
>>
>> pnfs_get_layout_done passes session and delay errors. Handle them wi=
th the
>> generic error handlers.
>>
>> Signed-off-by: Andy Adamson <andros@netapp.com>
>> ---
>> =A0fs/nfs/nfs4proc.c | =A0 45 +++++++++++++++++++++++++++++---------=
-------
>> =A01 files changed, 29 insertions(+), 16 deletions(-)
>>
>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>> index 35e8a56..6283996 100644
>> --- a/fs/nfs/nfs4proc.c
>> +++ b/fs/nfs/nfs4proc.c
>> @@ -5446,6 +5446,10 @@ static void nfs4_pnfs_layoutget_done(struct r=
pc_task *task, void *calldata)
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 return;
>>
>> =A0 =A0 =A0 pnfs_get_layout_done(lgp, task->tk_status);
>> +
>> + =A0 =A0 if (nfs4_async_handle_error(task, server, NULL, NULL) =3D=3D=
 -EAGAIN)
>> + =A0 =A0 =A0 =A0 =A0 =A0 nfs_restart_rpc(task, server->nfs_client);
>> +
>
> Andy high.
>
> In pnfs_get_layout_done() it is working hard to do "lgp->status =3D X=
XX"
>
> I never understood, what gets to look at that later on?

HI Boaz

_pnfs4_proc_layoutget will return it if it is set, then send_layoutget
returns it to pnfs_update_layout.

-->Andy

>
> Thanks
> Boaz
>
>> =A0 =A0 =A0 dprintk("<-- %s\n", __func__);
>> =A0}
>>
>> @@ -5471,10 +5475,9 @@ static const struct rpc_call_ops nfs4_pnfs_la=
youtget_call_ops =3D {
>> =A0 * and deal with retries.
>> =A0 * Currently we can't since we release lgp and its contents.
>> =A0 */
>> -int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
>> +static int _pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
>> =A0{
>> - =A0 =A0 struct inode *ino =3D lgp->args.inode;
>> - =A0 =A0 struct nfs_server *server =3D NFS_SERVER(ino);
>> + =A0 =A0 struct nfs_server *server =3D NFS_SERVER(lgp->args.inode);
>> =A0 =A0 =A0 struct rpc_task *task;
>> =A0 =A0 =A0 struct rpc_message msg =3D {
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 .rpc_proc =3D &nfs4_procedures[NFSPROC4_=
CLNT_PNFS_LAYOUTGET],
>> @@ -5488,35 +5491,45 @@ int pnfs4_proc_layoutget(struct nfs4_pnfs_la=
youtget *lgp)
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 .callback_data =3D lgp,
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 .flags =3D RPC_TASK_ASYNC,
>> =A0 =A0 =A0 };
>> - =A0 =A0 int status;
>> + =A0 =A0 int status =3D 0;
>>
>> =A0 =A0 =A0 dprintk("--> %s\n", __func__);
>>
>> =A0 =A0 =A0 lgp->res.layout.buf =3D (void *)__get_free_page(GFP_NOFS=
);
>> =A0 =A0 =A0 if (lgp->res.layout.buf =3D=3D NULL) {
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 nfs4_pnfs_layoutget_release(lgp);
>> - =A0 =A0 =A0 =A0 =A0 =A0 status =3D -ENOMEM;
>> - =A0 =A0 =A0 =A0 =A0 =A0 goto out;
>> + =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM;
>> =A0 =A0 =A0 }
>>
>> =A0 =A0 =A0 lgp->res.seq_res.sr_slotid =3D NFS4_MAX_SLOT_TABLE;
>> =A0 =A0 =A0 task =3D rpc_run_task(&task_setup_data);
>> - =A0 =A0 if (IS_ERR(task)) {
>> - =A0 =A0 =A0 =A0 =A0 =A0 status =3D PTR_ERR(task);
>> - =A0 =A0 =A0 =A0 =A0 =A0 goto out;
>> - =A0 =A0 }
>> + =A0 =A0 if (IS_ERR(task))
>> + =A0 =A0 =A0 =A0 =A0 =A0 return PTR_ERR(task);
>> =A0 =A0 =A0 status =3D nfs4_wait_for_completion_rpc_task(task);
>> - =A0 =A0 if (status =3D=3D 0) {
>> - =A0 =A0 =A0 =A0 =A0 =A0 status =3D lgp->status;
>> - =A0 =A0 =A0 =A0 =A0 =A0 if (status =3D=3D 0)
>> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 status =3D pnfs_layout_pro=
cess(lgp);
>> - =A0 =A0 }
>> - =A0 =A0 rpc_put_task(task);
>> + =A0 =A0 if (status !=3D 0)
>> + =A0 =A0 =A0 =A0 =A0 =A0 goto out;
>> + =A0 =A0 status =3D lgp->status;
>> + =A0 =A0 if (status !=3D 0)
>> + =A0 =A0 =A0 =A0 =A0 =A0 goto out;
>> + =A0 =A0 status =3D pnfs_layout_process(lgp);
>> =A0out:
>> + =A0 =A0 rpc_put_task(task);
>> =A0 =A0 =A0 dprintk("<-- %s status=3D%d\n", __func__, status);
>> =A0 =A0 =A0 return status;
>> =A0}
>>
>> +int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp)
>> +{
>> + =A0 =A0 struct nfs_server *server =3D NFS_SERVER(lgp->args.inode);
>> + =A0 =A0 struct nfs4_exception exception =3D { };
>> + =A0 =A0 int err;
>> + =A0 =A0 do {
>> + =A0 =A0 =A0 =A0 =A0 =A0 err =3D nfs4_handle_exception(server, _pnf=
s4_proc_layoutget(lgp),
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 =A0 &exception);
>> + =A0 =A0 } while (exception.retry);
>> + =A0 =A0 return err;
>> +}
>> +
>> =A0static void pnfs_layoutcommit_prepare(struct rpc_task *task, void=
 *data)
>> =A0{
>> =A0 =A0 =A0 struct pnfs_layoutcommit_data *ldata =3D
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" =
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[Benny Halevy]

On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote:
> Responded to comments, added a 2 cleanup patchses
> 
> Plus some code cleanup
> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch
> 
> and some bug fixes
> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch
> 
> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch
> was replaced by:
> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
> 
> 
> Remove unused (by file layout) encode_layoutreturn io operation
> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch
> 
> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch
> 
> Note: pnfs4_proc_layoutget is only called by send_layout() which prints
> the status.
> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch
> 
> Add back encode_layoutreturn io operation
> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch
> 
> 
> New patches:
> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch
> 
> This gets rid of a frame stack warning;
> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch
> 
> Testing:
> ---------
> 
> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
> Passes Connectathon tests
> 
> Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the
> pyNFS server and the testclient framework.
> 
> Still todo:
> 
> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and
> layoutcommit do not pass nfs_stste to the error handlers.
> 
> Handle NFS4ERR_BAD_LAYOUT.
> 
> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.
> 
> -->Andy

Andy, I've hit
	BUG_ON(lo->refcount <= 0);
in put_layout() with this patchset.
I'm not sure if it introduced it or not, still investigating...

Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: layout.refcount 1
Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------
Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341!
Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/initstate
Jun 24 12:07:26 tl2 kernel: CPU 1 
Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore snd_page_alloc button video output rng_core sg cdrom ata_generic ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
Jun 24 12:07:26 tl2 kernel:
Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527
Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>]  [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20  EFLAGS: 00010246
Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: ffff8800704b6b78 RCX: 0000000000000066
Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: ffffea0001b931a8 RDI: ffff8800704b6b78
Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: 0000000000000000 R09: ffff88007356a500
Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: 0000000000000003 R12: ffff8800704b69a8
Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: ffff88007356a508 R15: ffff88007356a590
Jun 24 12:07:26 tl2 kernel: FS:  0000000000000000(0000) GS:ffff880001a80000(0000) knlGS:0000000000000000
Jun 24 12:07:26 tl2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: 0000000001698000 CR4: 00000000000406e0
Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo ffff88007525c000, task ffff88007d988000)
Jun 24 12:07:26 tl2 kernel: Stack:
Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 ffff88007525dd60 ffffffffa05d203f
Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 ffff880073854f00 ffffffffa05d5880
Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c ffff88007525dd90 ffff88007356a500
Jun 24 12:07:26 tl2 kernel: Call Trace:
Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release+0x43/0x68 [nfs]
Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs]
Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] rpc_release_calldata+0x17/0x19 [sunrpc]
Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/0x66 [sunrpc]
Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task+0x98/0x9c [sunrpc]
Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute+0x205/0x212 [sunrpc]
Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule+0x15/0x17 [sunrpc]
Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread+0x1aa/0x23b
Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? rpc_async_schedule+0x0/0x17 [sunrpc]
Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? autoremove_wake_function+0x0/0x39
Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? spin_unlock_irqrestore+0xe/0x10
Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread+0x0/0x23b
Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87
Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] kernel_thread_helper+0x4/0x10
Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87
Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? kernel_thread_helper+0x0/0x10
Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67 48 8b 82 48 03 00 00 f6 05 
Jun 24 12:07:26 tl2 kernel: RIP  [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20>
Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]---

Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[William A. (Andy) Adamson]

OK - I'll look into it.

Sorry I missed today's pNFS call.

-->Andy

On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com> wrote:
> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote:
>> Responded to comments, added a 2 cleanup patchses
>>
>> Plus some code cleanup
>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch
>>
>> and some bug fixes
>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch
>>
>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch
>> was replaced by:
>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
>>
>>
>> Remove unused (by file layout) encode_layoutreturn io operation
>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch
>>
>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch
>>
>> Note: pnfs4_proc_layoutget is only called by send_layout() which prints
>> the status.
>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch
>>
>> Add back encode_layoutreturn io operation
>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch
>>
>>
>> New patches:
>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch
>>
>> This gets rid of a frame stack warning;
>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch
>>
>> Testing:
>> ---------
>>
>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
>> Passes Connectathon tests
>>
>> Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the
>> pyNFS server and the testclient framework.
>>
>> Still todo:
>>
>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and
>> layoutcommit do not pass nfs_stste to the error handlers.
>>
>> Handle NFS4ERR_BAD_LAYOUT.
>>
>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.
>>
>> -->Andy
>
> Andy, I've hit
>        BUG_ON(lo->refcount <= 0);
> in put_layout() with this patchset.
> I'm not sure if it introduced it or not, still investigating...
>
> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: layout.refcount 1
> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------
> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341!
> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/initstate
> Jun 24 12:07:26 tl2 kernel: CPU 1
> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore snd_page_alloc button video output rng_core sg cdrom ata_generic ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
> Jun 24 12:07:26 tl2 kernel:
> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527
> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>]  [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20  EFLAGS: 00010246
> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: ffff8800704b6b78 RCX: 0000000000000066
> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: ffffea0001b931a8 RDI: ffff8800704b6b78
> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: 0000000000000000 R09: ffff88007356a500
> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: 0000000000000003 R12: ffff8800704b69a8
> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: ffff88007356a508 R15: ffff88007356a590
> Jun 24 12:07:26 tl2 kernel: FS:  0000000000000000(0000) GS:ffff880001a80000(0000) knlGS:0000000000000000
> Jun 24 12:07:26 tl2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: 0000000001698000 CR4: 00000000000406e0
> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo ffff88007525c000, task ffff88007d988000)
> Jun 24 12:07:26 tl2 kernel: Stack:
> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 ffff88007525dd60 ffffffffa05d203f
> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 ffff880073854f00 ffffffffa05d5880
> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c ffff88007525dd90 ffff88007356a500
> Jun 24 12:07:26 tl2 kernel: Call Trace:
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release+0x43/0x68 [nfs]
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs]
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] rpc_release_calldata+0x17/0x19 [sunrpc]
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/0x66 [sunrpc]
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task+0x98/0x9c [sunrpc]
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute+0x205/0x212 [sunrpc]
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule+0x15/0x17 [sunrpc]
> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread+0x1aa/0x23b
> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? rpc_async_schedule+0x0/0x17 [sunrpc]
> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? autoremove_wake_function+0x0/0x39
> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? spin_unlock_irqrestore+0xe/0x10
> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread+0x0/0x23b
> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87
> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] kernel_thread_helper+0x4/0x10
> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87
> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? kernel_thread_helper+0x0/0x10
> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67 48 8b 82 48 03 00 00 f6 05
> Jun 24 12:07:26 tl2 kernel: RIP  [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20>
> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]---
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[Andy Adamson]

Hi Benny

I have not been able to reproduce this BUG. I've tried against the  
files pyNFS server with return_on_close False as well a True, and  
against a GFS2/pNFS cluster with write layouts turned on.

Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch  
calls put_lseg when I/O to a DS fails. I tested this using the pyNFS  
files layout server and blocking the DS with iptables. I think this is  
the only change in this patch set that would affect the refcounting.

Are you able to reproduce the BUG?

-->Andy

On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote:

> OK - I'll look into it.
>
> Sorry I missed today's pNFS call.
>
> -->Andy
>
> On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com>  
> wrote:
>> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote:
>>> Responded to comments, added a 2 cleanup patchses
>>>
>>> Plus some code cleanup
>>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch
>>>
>>> and some bug fixes
>>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch
>>>
>>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the- 
>>> MDS.patch
>>> was replaced by:
>>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
>>>
>>>
>>> Remove unused (by file layout) encode_layoutreturn io operation
>>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
>>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch
>>>
>>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch
>>>
>>> Note: pnfs4_proc_layoutget is only called by send_layout() which  
>>> prints
>>> the status.
>>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch
>>>
>>> Add back encode_layoutreturn io operation
>>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch
>>>
>>>
>>> New patches:
>>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch
>>>
>>> This gets rid of a frame stack warning;
>>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch
>>>
>>> Testing:
>>> ---------
>>>
>>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
>>> Passes Connectathon tests
>>>
>>> Tested layoutget and layoutreturn recovery from  
>>> NFS4ERR_DEAD_SESSION with the
>>> pyNFS server and the testclient framework.
>>>
>>> Still todo:
>>>
>>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn,  
>>> layoutget, and
>>> layoutcommit do not pass nfs_stste to the error handlers.
>>>
>>> Handle NFS4ERR_BAD_LAYOUT.
>>>
>>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.
>>>
>>> -->Andy
>>
>> Andy, I've hit
>>       BUG_ON(lo->refcount <= 0);
>> in put_layout() with this patchset.
>> I'm not sure if it introduced it or not, still investigating...
>>
>> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING:  
>> layout.refcount 1
>> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------
>> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/ 
>> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341!
>> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP  
>> DEBUG_PAGEALLOC
>> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/ 
>> initstate
>> Jun 24 12:07:26 tl2 kernel: CPU 1
>> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd  
>> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4  
>> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi  
>> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror  
>> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm  
>> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel  
>> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core  
>> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore  
>> snd_page_alloc button video output rng_core sg cdrom ata_generic  
>> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd  
>> ohci_hcd ehci_hcd [last unloaded: microcode]
>> Jun 24 12:07:26 tl2 kernel:
>> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted  
>> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527
>> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>]   
>> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
>> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20  EFLAGS:  
>> 00010246
>> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX:  
>> ffff8800704b6b78 RCX: 0000000000000066
>> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI:  
>> ffffea0001b931a8 RDI: ffff8800704b6b78
>> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08:  
>> 0000000000000000 R09: ffff88007356a500
>> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11:  
>> 0000000000000003 R12: ffff8800704b69a8
>> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14:  
>> ffff88007356a508 R15: ffff88007356a590
>> Jun 24 12:07:26 tl2 kernel: FS:  0000000000000000(0000)  
>> GS:ffff880001a80000(0000) knlGS:0000000000000000
>> Jun 24 12:07:26 tl2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:  
>> 000000008005003b
>> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3:  
>> 0000000001698000 CR4: 00000000000406e0
>> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1:  
>> 0000000000000000 DR2: 0000000000000000
>> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6:  
>> 00000000ffff0ff0 DR7: 0000000000000400
>> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo  
>> ffff88007525c000, task ffff88007d988000)
>> Jun 24 12:07:26 tl2 kernel: Stack:
>> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8  
>> ffff88007525dd60 ffffffffa05d203f
>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18  
>> ffff880073854f00 ffffffffa05d5880
>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c  
>> ffff88007525dd90 ffff88007356a500
>> Jun 24 12:07:26 tl2 kernel: Call Trace:
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release 
>> +0x43/0x68 [nfs]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>]  
>> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>]  
>> rpc_release_calldata+0x17/0x19 [sunrpc]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/ 
>> 0x66 [sunrpc]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task 
>> +0x98/0x9c [sunrpc]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute 
>> +0x205/0x212 [sunrpc]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule 
>> +0x15/0x17 [sunrpc]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread 
>> +0x1aa/0x23b
>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ?  
>> rpc_async_schedule+0x0/0x17 [sunrpc]
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ?  
>> autoremove_wake_function+0x0/0x39
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ?  
>> spin_unlock_irqrestore+0xe/0x10
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread 
>> +0x0/0x23b
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>]  
>> kernel_thread_helper+0x4/0x10
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87
>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ?  
>> kernel_thread_helper+0x0/0x10
>> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24  
>> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04  
>> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67  
>> 48 8b 82 48 03 00 00 f6 05
>> Jun 24 12:07:26 tl2 kernel: RIP  [<ffffffffa05d0ea4>] put_layout 
>> +0x2f/0xa7 [nfs]
>> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20>
>> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]---
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux- 
>> nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"  
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[Benny Halevy]

On Jun. 28, 2010, 19:44 +0300, Andy Adamson <andros@netapp.com> wrote:
> Hi Benny
> 
> I have not been able to reproduce this BUG. I've tried against the  
> files pyNFS server with return_on_close False as well a True, and  
> against a GFS2/pNFS cluster with write layouts turned on.
> 
> Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch  
> calls put_lseg when I/O to a DS fails. I tested this using the pyNFS  
> files layout server and blocking the DS with iptables. I think this is  
> the only change in this patch set that would affect the refcounting.
> 
> Are you able to reproduce the BUG?

The easiest way I found to reproduce this bug is running the cthon tests
on a locally mounted file system exported over PNFSD_LOCAL_EXPORT.
The test machine is a dual core SMP machine.
Are you testing over a VM?  Is it uni-processor?

Benny

> 
> -->Andy
> 
> On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote:
> 
>> OK - I'll look into it.
>>
>> Sorry I missed today's pNFS call.
>>
>> -->Andy
>>
>> On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com>  
>> wrote:
>>> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote:
>>>> Responded to comments, added a 2 cleanup patchses
>>>>
>>>> Plus some code cleanup
>>>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch
>>>>
>>>> and some bug fixes
>>>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch
>>>>
>>>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the- 
>>>> MDS.patch
>>>> was replaced by:
>>>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
>>>>
>>>>
>>>> Remove unused (by file layout) encode_layoutreturn io operation
>>>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
>>>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch
>>>>
>>>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch
>>>>
>>>> Note: pnfs4_proc_layoutget is only called by send_layout() which  
>>>> prints
>>>> the status.
>>>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch
>>>>
>>>> Add back encode_layoutreturn io operation
>>>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch
>>>>
>>>>
>>>> New patches:
>>>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch
>>>>
>>>> This gets rid of a frame stack warning;
>>>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch
>>>>
>>>> Testing:
>>>> ---------
>>>>
>>>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
>>>> Passes Connectathon tests
>>>>
>>>> Tested layoutget and layoutreturn recovery from  
>>>> NFS4ERR_DEAD_SESSION with the
>>>> pyNFS server and the testclient framework.
>>>>
>>>> Still todo:
>>>>
>>>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn,  
>>>> layoutget, and
>>>> layoutcommit do not pass nfs_stste to the error handlers.
>>>>
>>>> Handle NFS4ERR_BAD_LAYOUT.
>>>>
>>>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.
>>>>
>>>> -->Andy
>>>
>>> Andy, I've hit
>>>       BUG_ON(lo->refcount <= 0);
>>> in put_layout() with this patchset.
>>> I'm not sure if it introduced it or not, still investigating...
>>>
>>> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING:  
>>> layout.refcount 1
>>> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------
>>> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/ 
>>> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341!
>>> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP  
>>> DEBUG_PAGEALLOC
>>> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/ 
>>> initstate
>>> Jun 24 12:07:26 tl2 kernel: CPU 1
>>> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd  
>>> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4  
>>> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi  
>>> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror  
>>> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm  
>>> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel  
>>> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core  
>>> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore  
>>> snd_page_alloc button video output rng_core sg cdrom ata_generic  
>>> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd  
>>> ohci_hcd ehci_hcd [last unloaded: microcode]
>>> Jun 24 12:07:26 tl2 kernel:
>>> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted  
>>> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527
>>> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>]   
>>> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
>>> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20  EFLAGS:  
>>> 00010246
>>> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX:  
>>> ffff8800704b6b78 RCX: 0000000000000066
>>> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI:  
>>> ffffea0001b931a8 RDI: ffff8800704b6b78
>>> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08:  
>>> 0000000000000000 R09: ffff88007356a500
>>> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11:  
>>> 0000000000000003 R12: ffff8800704b69a8
>>> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14:  
>>> ffff88007356a508 R15: ffff88007356a590
>>> Jun 24 12:07:26 tl2 kernel: FS:  0000000000000000(0000)  
>>> GS:ffff880001a80000(0000) knlGS:0000000000000000
>>> Jun 24 12:07:26 tl2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:  
>>> 000000008005003b
>>> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3:  
>>> 0000000001698000 CR4: 00000000000406e0
>>> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1:  
>>> 0000000000000000 DR2: 0000000000000000
>>> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6:  
>>> 00000000ffff0ff0 DR7: 0000000000000400
>>> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo  
>>> ffff88007525c000, task ffff88007d988000)
>>> Jun 24 12:07:26 tl2 kernel: Stack:
>>> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8  
>>> ffff88007525dd60 ffffffffa05d203f
>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18  
>>> ffff880073854f00 ffffffffa05d5880
>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c  
>>> ffff88007525dd90 ffff88007356a500
>>> Jun 24 12:07:26 tl2 kernel: Call Trace:
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release 
>>> +0x43/0x68 [nfs]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>]  
>>> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>]  
>>> rpc_release_calldata+0x17/0x19 [sunrpc]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/ 
>>> 0x66 [sunrpc]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task 
>>> +0x98/0x9c [sunrpc]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute 
>>> +0x205/0x212 [sunrpc]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule 
>>> +0x15/0x17 [sunrpc]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread 
>>> +0x1aa/0x23b
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ?  
>>> rpc_async_schedule+0x0/0x17 [sunrpc]
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ?  
>>> autoremove_wake_function+0x0/0x39
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ?  
>>> spin_unlock_irqrestore+0xe/0x10
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread 
>>> +0x0/0x23b
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>]  
>>> kernel_thread_helper+0x4/0x10
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87
>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ?  
>>> kernel_thread_helper+0x0/0x10
>>> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24  
>>> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04  
>>> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67  
>>> 48 8b 82 48 03 00 00 f6 05
>>> Jun 24 12:07:26 tl2 kernel: RIP  [<ffffffffa05d0ea4>] put_layout 
>>> +0x2f/0xa7 [nfs]
>>> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20>
>>> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]---
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux- 
>>> nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs"  
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[William A. (Andy) Adamson]

On Mon, Jun 28, 2010 at 2:53 PM, Benny Halevy <bhalevy@panasas.com> wro=
te:
> On Jun. 28, 2010, 19:44 +0300, Andy Adamson <andros@netapp.com> wrote=
:
>> Hi Benny
>>
>> I have not been able to reproduce this BUG. I've tried against the
>> files pyNFS server with return_on_close False as well a True, and
>> against a GFS2/pNFS cluster with write layouts turned on.
>>
>> Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
>> calls put_lseg when I/O to a DS fails. I tested this using the pyNFS
>> files layout server and blocking the DS with iptables. I think this =
is
>> the only change in this patch set that would affect the refcounting.
>>
>> Are you able to reproduce the BUG?
>
> The easiest way I found to reproduce this bug is running the cthon te=
sts
> on a locally mounted file system exported over PNFSD_LOCAL_EXPORT.
> The test machine is a dual core SMP machine.
> Are you testing over a VM? =A0Is it uni-processor?

Its a VM with one processor, but with SMP support turned on in the
kernel. I just added a processor and will try re-running tests.

-->Andy

>
> Benny
>
>>
>> -->Andy
>>
>> On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote:
>>
>>> OK - I'll look into it.
>>>
>>> Sorry I missed today's pNFS call.
>>>
>>> -->Andy
>>>
>>> On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com>
>>> wrote:
>>>> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote:
>>>>> Responded to comments, added a 2 cleanup patchses
>>>>>
>>>>> Plus some code cleanup
>>>>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch
>>>>>
>>>>> and some bug fixes
>>>>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch
>>>>>
>>>>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-th=
e-
>>>>> MDS.patch
>>>>> was replaced by:
>>>>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
>>>>>
>>>>>
>>>>> Remove unused (by file layout) encode_layoutreturn io operation
>>>>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
>>>>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch
>>>>>
>>>>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch
>>>>>
>>>>> Note: pnfs4_proc_layoutget is only called by send_layout() which
>>>>> prints
>>>>> the status.
>>>>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch
>>>>>
>>>>> Add back encode_layoutreturn io operation
>>>>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch
>>>>>
>>>>>
>>>>> New patches:
>>>>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch
>>>>>
>>>>> This gets rid of a frame stack warning;
>>>>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch
>>>>>
>>>>> Testing:
>>>>> ---------
>>>>>
>>>>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
>>>>> Passes Connectathon tests
>>>>>
>>>>> Tested layoutget and layoutreturn recovery from
>>>>> NFS4ERR_DEAD_SESSION with the
>>>>> pyNFS server and the testclient framework.
>>>>>
>>>>> Still todo:
>>>>>
>>>>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn,
>>>>> layoutget, and
>>>>> layoutcommit do not pass nfs_stste to the error handlers.
>>>>>
>>>>> Handle NFS4ERR_BAD_LAYOUT.
>>>>>
>>>>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.
>>>>>
>>>>> -->Andy
>>>>
>>>> Andy, I've hit
>>>> =A0 =A0 =A0 BUG_ON(lo->refcount <=3D 0);
>>>> in put_layout() with this patchset.
>>>> I'm not sure if it introduced it or not, still investigating...
>>>>
>>>> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING:
>>>> layout.refcount 1
>>>> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------
>>>> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy=
/
>>>> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341!
>>>> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP
>>>> DEBUG_PAGEALLOC
>>>> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/
>>>> initstate
>>>> Jun 24 12:07:26 tl2 kernel: CPU 1
>>>> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfs=
d
>>>> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4
>>>> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
>>>> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror
>>>> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm
>>>> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel
>>>> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core
>>>> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore
>>>> snd_page_alloc button video output rng_core sg cdrom ata_generic
>>>> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd
>>>> ohci_hcd ehci_hcd [last unloaded: microcode]
>>>> Jun 24 12:07:26 tl2 kernel:
>>>> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted
>>>> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527
>>>> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>]
>>>> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
>>>> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 =A0EFLAGS:
>>>> 00010246
>>>> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX:
>>>> ffff8800704b6b78 RCX: 0000000000000066
>>>> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI:
>>>> ffffea0001b931a8 RDI: ffff8800704b6b78
>>>> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08:
>>>> 0000000000000000 R09: ffff88007356a500
>>>> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11:
>>>> 0000000000000003 R12: ffff8800704b69a8
>>>> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14:
>>>> ffff88007356a508 R15: ffff88007356a590
>>>> Jun 24 12:07:26 tl2 kernel: FS: =A00000000000000000(0000)
>>>> GS:ffff880001a80000(0000) knlGS:0000000000000000
>>>> Jun 24 12:07:26 tl2 kernel: CS: =A00010 DS: 0000 ES: 0000 CR0:
>>>> 000000008005003b
>>>> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3:
>>>> 0000000001698000 CR4: 00000000000406e0
>>>> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1:
>>>> 0000000000000000 DR2: 0000000000000000
>>>> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6:
>>>> 00000000ffff0ff0 DR7: 0000000000000400
>>>> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinf=
o
>>>> ffff88007525c000, task ffff88007d988000)
>>>> Jun 24 12:07:26 tl2 kernel: Stack:
>>>> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8
>>>> ffff88007525dd60 ffffffffa05d203f
>>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18
>>>> ffff880073854f00 ffffffffa05d5880
>>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c
>>>> ffff88007525dd90 ffff88007356a500
>>>> Jun 24 12:07:26 tl2 kernel: Call Trace:
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_relea=
se
>>>> +0x43/0x68 [nfs]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>]
>>>> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>]
>>>> rpc_release_calldata+0x17/0x19 [sunrpc]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5=
e/
>>>> 0x66 [sunrpc]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task
>>>> +0x98/0x9c [sunrpc]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute
>>>> +0x205/0x212 [sunrpc]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedul=
e
>>>> +0x15/0x17 [sunrpc]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread
>>>> +0x1aa/0x23b
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ?
>>>> rpc_async_schedule+0x0/0x17 [sunrpc]
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ?
>>>> autoremove_wake_function+0x0/0x39
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ?
>>>> spin_unlock_irqrestore+0xe/0x10
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread
>>>> +0x0/0x23b
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>]
>>>> kernel_thread_helper+0x4/0x10
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x8=
7
>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ?
>>>> kernel_thread_helper+0x0/0x10
>>>> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24
>>>> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04
>>>> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 6=
7
>>>> 48 8b 82 48 03 00 00 f6 05
>>>> Jun 24 12:07:26 tl2 kernel: RIP =A0[<ffffffffa05d0ea4>] put_layout
>>>> +0x2f/0xa7 [nfs]
>>>> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20>
>>>> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]---
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-
>>>> nfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.ht=
ml
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs=
"
>>> in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm=
l
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs"=
 in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" =
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[William A. (Andy) Adamson]

On Mon, Jun 28, 2010 at 3:22 PM, William A. (Andy) Adamson
<androsadamson@gmail.com> wrote:
> On Mon, Jun 28, 2010 at 2:53 PM, Benny Halevy <bhalevy@panasas.com> w=
rote:
>> On Jun. 28, 2010, 19:44 +0300, Andy Adamson <andros@netapp.com> wrot=
e:
>>> Hi Benny
>>>
>>> I have not been able to reproduce this BUG. I've tried against the
>>> files pyNFS server with return_on_close False as well a True, and
>>> against a GFS2/pNFS cluster with write layouts turned on.
>>>
>>> Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patc=
h
>>> calls put_lseg when I/O to a DS fails. I tested this using the pyNF=
S
>>> files layout server and blocking the DS with iptables. I think this=
 is
>>> the only change in this patch set that would affect the refcounting=
=2E
>>>
>>> Are you able to reproduce the BUG?
>>
>> The easiest way I found to reproduce this bug is running the cthon t=
ests
>> on a locally mounted file system exported over PNFSD_LOCAL_EXPORT.
>> The test machine is a dual core SMP machine.
>> Are you testing over a VM? =A0Is it uni-processor?
>
> Its a VM with one processor, but with SMP support turned on in the
> kernel. I just added a processor and will try re-running tests.

Added a processor  - all cthon tests succeeded. Just to be clear, I'm
testing the client pnfs-submit branch.

-->Andy

>
> -->Andy
>
>>
>> Benny
>>
>>>
>>> -->Andy
>>>
>>> On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote:
>>>
>>>> OK - I'll look into it.
>>>>
>>>> Sorry I missed today's pNFS call.
>>>>
>>>> -->Andy
>>>>
>>>> On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com=
>
>>>> wrote:
>>>>> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote:
>>>>>> Responded to comments, added a 2 cleanup patchses
>>>>>>
>>>>>> Plus some code cleanup
>>>>>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch
>>>>>>
>>>>>> and some bug fixes
>>>>>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch
>>>>>>
>>>>>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-t=
he-
>>>>>> MDS.patch
>>>>>> was replaced by:
>>>>>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
>>>>>>
>>>>>>
>>>>>> Remove unused (by file layout) encode_layoutreturn io operation
>>>>>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
>>>>>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch
>>>>>>
>>>>>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch
>>>>>>
>>>>>> Note: pnfs4_proc_layoutget is only called by send_layout() which
>>>>>> prints
>>>>>> the status.
>>>>>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch
>>>>>>
>>>>>> Add back encode_layoutreturn io operation
>>>>>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch
>>>>>>
>>>>>>
>>>>>> New patches:
>>>>>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch
>>>>>>
>>>>>> This gets rid of a frame stack warning;
>>>>>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch
>>>>>>
>>>>>> Testing:
>>>>>> ---------
>>>>>>
>>>>>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
>>>>>> Passes Connectathon tests
>>>>>>
>>>>>> Tested layoutget and layoutreturn recovery from
>>>>>> NFS4ERR_DEAD_SESSION with the
>>>>>> pyNFS server and the testclient framework.
>>>>>>
>>>>>> Still todo:
>>>>>>
>>>>>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn,
>>>>>> layoutget, and
>>>>>> layoutcommit do not pass nfs_stste to the error handlers.
>>>>>>
>>>>>> Handle NFS4ERR_BAD_LAYOUT.
>>>>>>
>>>>>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.
>>>>>>
>>>>>> -->Andy
>>>>>
>>>>> Andy, I've hit
>>>>> =A0 =A0 =A0 BUG_ON(lo->refcount <=3D 0);
>>>>> in put_layout() with this patchset.
>>>>> I'm not sure if it introduced it or not, still investigating...
>>>>>
>>>>> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING:
>>>>> layout.refcount 1
>>>>> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------
>>>>> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalev=
y/
>>>>> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341!
>>>>> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP
>>>>> DEBUG_PAGEALLOC
>>>>> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/
>>>>> initstate
>>>>> Jun 24 12:07:26 tl2 kernel: CPU 1
>>>>> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nf=
sd
>>>>> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4
>>>>> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
>>>>> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror
>>>>> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm
>>>>> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel
>>>>> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core
>>>>> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore
>>>>> snd_page_alloc button video output rng_core sg cdrom ata_generic
>>>>> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd
>>>>> ohci_hcd ehci_hcd [last unloaded: microcode]
>>>>> Jun 24 12:07:26 tl2 kernel:
>>>>> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted
>>>>> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527
>>>>> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>]
>>>>> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs]
>>>>> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 =A0EFLAGS:
>>>>> 00010246
>>>>> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX:
>>>>> ffff8800704b6b78 RCX: 0000000000000066
>>>>> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI:
>>>>> ffffea0001b931a8 RDI: ffff8800704b6b78
>>>>> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08:
>>>>> 0000000000000000 R09: ffff88007356a500
>>>>> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11:
>>>>> 0000000000000003 R12: ffff8800704b69a8
>>>>> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14:
>>>>> ffff88007356a508 R15: ffff88007356a590
>>>>> Jun 24 12:07:26 tl2 kernel: FS: =A00000000000000000(0000)
>>>>> GS:ffff880001a80000(0000) knlGS:0000000000000000
>>>>> Jun 24 12:07:26 tl2 kernel: CS: =A00010 DS: 0000 ES: 0000 CR0:
>>>>> 000000008005003b
>>>>> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3:
>>>>> 0000000001698000 CR4: 00000000000406e0
>>>>> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1:
>>>>> 0000000000000000 DR2: 0000000000000000
>>>>> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6:
>>>>> 00000000ffff0ff0 DR7: 0000000000000400
>>>>> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadin=
fo
>>>>> ffff88007525c000, task ffff88007d988000)
>>>>> Jun 24 12:07:26 tl2 kernel: Stack:
>>>>> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8
>>>>> ffff88007525dd60 ffffffffa05d203f
>>>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18
>>>>> ffff880073854f00 ffffffffa05d5880
>>>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c
>>>>> ffff88007525dd90 ffff88007356a500
>>>>> Jun 24 12:07:26 tl2 kernel: Call Trace:
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_rele=
ase
>>>>> +0x43/0x68 [nfs]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>]
>>>>> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>]
>>>>> rpc_release_calldata+0x17/0x19 [sunrpc]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x=
5e/
>>>>> 0x66 [sunrpc]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task
>>>>> +0x98/0x9c [sunrpc]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute
>>>>> +0x205/0x212 [sunrpc]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedu=
le
>>>>> +0x15/0x17 [sunrpc]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread
>>>>> +0x1aa/0x23b
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ?
>>>>> rpc_async_schedule+0x0/0x17 [sunrpc]
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ?
>>>>> autoremove_wake_function+0x0/0x39
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ?
>>>>> spin_unlock_irqrestore+0xe/0x10
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread
>>>>> +0x0/0x23b
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x8=
7
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>]
>>>>> kernel_thread_helper+0x4/0x10
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x=
87
>>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ?
>>>>> kernel_thread_helper+0x0/0x10
>>>>> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 2=
4
>>>>> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04
>>>>> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 =
67
>>>>> 48 8b 82 48 03 00 00 f6 05
>>>>> Jun 24 12:07:26 tl2 kernel: RIP =A0[<ffffffffa05d0ea4>] put_layou=
t
>>>>> +0x2f/0xa7 [nfs]
>>>>> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20>
>>>>> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]---
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-
>>>>> nfs" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.h=
tml
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nf=
s"
>>>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.ht=
ml
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs=
" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm=
l
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs"=
 in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>>
>

Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2

[Benny Halevy]

On 2010-06-23 22:21, andros@netapp.com wrote:On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote:
> Responded to comments, added a 2 cleanup patchses
> 
> Plus some code cleanup
> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch
> 
> and some bug fixes
> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch
> 
> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch
> was replaced by:
> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch
> 

All but patch 3/10 (as per your request) are merged at
pnfs-all-2.6.35-rc3-2010-07-01

Thanks!

Benny

> 
> Remove unused (by file layout) encode_layoutreturn io operation
> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch
> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch
> 
> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch
> 
> Note: pnfs4_proc_layoutget is only called by send_layout() which prints
> the status.
> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch
> 
> Add back encode_layoutreturn io operation
> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch
> 
> 
> New patches:
> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch
> 
> This gets rid of a frame stack warning;
> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch
> 
> Testing:
> ---------
> 
> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS
> Passes Connectathon tests
> 
> Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the
> pyNFS server and the testclient framework.
> 
> Still todo:
> 
> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and
> layoutcommit do not pass nfs_stste to the error handlers.
> 
> Handle NFS4ERR_BAD_LAYOUT.
> 
> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests.
> 
> -->Andy


-- 
Benny Halevy
Software Architect
Panasas, Inc.
bhalevy@panasas.com
Tel/Fax: +972-3-647-8340
Mobile: +972-54-802-8340

Panasas: The Leader in Parallel Storage
www.panasas.com