* [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 @ 2010-06-23 19:21 andros 2010-06-23 19:21 ` [PATCH 01/10] SQUASHME: pnfs-submit remove unused filelayout_mount_type andros ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs Responded to comments, added a 2 cleanup patchses Plus some code cleanup 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch and some bug fixes 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch was replaced by: 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch Remove unused (by file layout) encode_layoutreturn io operation 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch Note: pnfs4_proc_layoutget is only called by send_layout() which prints the status. 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch Add back encode_layoutreturn io operation 0008-pnfs-post-submit-restore-encode_layoutreturn.patch New patches: 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch This gets rid of a frame stack warning; 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch Testing: --------- CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS Passes Connectathon tests Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the pyNFS server and the testclient framework. Still todo: Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and layoutcommit do not pass nfs_stste to the error handlers. Handle NFS4ERR_BAD_LAYOUT. CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. -->Andy ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 01/10] SQUASHME: pnfs-submit remove unused filelayout_mount_type 2010-06-23 19:21 [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 02/10] SQUASHME pnfs-submit: pnfs_try_to_read, write, commit using freed memory andros 2010-06-24 13:14 ` [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 Benny Halevy 2010-07-01 18:27 ` Benny Halevy 2 siblings, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/nfs4filelayout.h | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h index 29e481d..de8391f 100644 --- a/fs/nfs/nfs4filelayout.h +++ b/fs/nfs/nfs4filelayout.h @@ -24,8 +24,6 @@ #define NFS4_PNFS_MAX_MULTI_CNT 64 /* 256 fit into a u8 stripe_index */ #define NFS4_PNFS_MAX_MULTI_DS 2 -#define FILE_MT(inode) ((struct filelayout_mount_type *) \ - (NFS_SERVER(inode)->pnfs_mountid->mountid)) #define FILE_DSADDR(lseg) (container_of(lseg->deviceid, \ struct nfs4_file_layout_dsaddr, \ deviceid)) @@ -76,10 +74,6 @@ struct nfs4_filelayout { u32 stripe_unit; }; -struct filelayout_mount_type { - struct super_block *fl_sb; -}; - extern struct nfs_fh * nfs4_fl_select_ds_fh(struct pnfs_layout_segment *lseg, loff_t offset); -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 02/10] SQUASHME pnfs-submit: pnfs_try_to_read, write, commit using freed memory 2010-06-23 19:21 ` [PATCH 01/10] SQUASHME: pnfs-submit remove unused filelayout_mount_type andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 03/10] SQUASHME pnfs-submit: clear page lseg on partial i/o andros 0 siblings, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> The nfs_read_data or nfs_write_data is freed during rpc call. Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/pnfs.h | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index 78776f3..c60eff6 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -105,13 +105,14 @@ static inline enum pnfs_try_status pnfs_try_to_read_data(struct nfs_read_data *data, const struct rpc_call_ops *call_ops) { + struct inode *inode = data->inode; enum pnfs_try_status ret; if (!data->req->wb_lseg) return PNFS_NOT_ATTEMPTED; ret = _pnfs_try_to_read_data(data, call_ops); if (ret == PNFS_ATTEMPTED) - nfs_inc_stats(data->inode, NFSIOS_PNFS_READ); + nfs_inc_stats(inode, NFSIOS_PNFS_READ); else _pnfs_clear_lseg_from_pages(&data->pages); return ret; @@ -122,13 +123,14 @@ pnfs_try_to_write_data(struct nfs_write_data *data, const struct rpc_call_ops *call_ops, int how) { + struct inode *inode = data->inode; enum pnfs_try_status ret; if (!data->req->wb_lseg) return PNFS_NOT_ATTEMPTED; ret = _pnfs_try_to_write_data(data, call_ops, how); if (ret == PNFS_ATTEMPTED) - nfs_inc_stats(data->inode, NFSIOS_PNFS_WRITE); + nfs_inc_stats(inode, NFSIOS_PNFS_WRITE); else _pnfs_clear_lseg_from_pages(&data->pages); return ret; @@ -139,6 +141,7 @@ pnfs_try_to_commit(struct nfs_write_data *data, const struct rpc_call_ops *call_ops, int how) { + struct inode *inode = data->inode; enum pnfs_try_status ret; /* Unlike in pnfs_try_to_write_data and pnfs_try_to_read_data, @@ -148,7 +151,7 @@ pnfs_try_to_commit(struct nfs_write_data *data, */ ret = _pnfs_try_to_commit(data, call_ops, how); if (ret == PNFS_ATTEMPTED) - nfs_inc_stats(data->inode, NFSIOS_PNFS_COMMIT); + nfs_inc_stats(inode, NFSIOS_PNFS_COMMIT); else _pnfs_clear_lseg_from_pages(&data->pages); return ret; -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 03/10] SQUASHME pnfs-submit: clear page lseg on partial i/o 2010-06-23 19:21 ` [PATCH 02/10] SQUASHME pnfs-submit: pnfs_try_to_read, write, commit using freed memory andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 04/10] SQUASHME pnfs-submit: remove encode_layoutreturn andros 0 siblings, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> partial reads or writes use a single page not the page list. Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/pnfs.h | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-) diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index c60eff6..70f1d97 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -91,10 +91,18 @@ static inline int pnfs_enabled_sb(struct nfs_server *nfss) return nfss->pnfs_curr_ld != NULL; } -static inline void _pnfs_clear_lseg_from_pages(struct list_head *head) +static inline void _pnfs_clear_lseg_from_pages(struct nfs_page *partial, + struct list_head *head) { struct nfs_page *req; + /* partial_read or partial_write */ + if (list_empty(head)) { + put_lseg(partial->wb_lseg); + partial->wb_lseg = NULL; + return; + } + /* full_read or full_write */ list_for_each_entry(req, head, wb_list) { put_lseg(req->wb_lseg); req->wb_lseg = NULL; @@ -114,7 +122,7 @@ pnfs_try_to_read_data(struct nfs_read_data *data, if (ret == PNFS_ATTEMPTED) nfs_inc_stats(inode, NFSIOS_PNFS_READ); else - _pnfs_clear_lseg_from_pages(&data->pages); + _pnfs_clear_lseg_from_pages(data->req, &data->pages); return ret; } @@ -132,7 +140,7 @@ pnfs_try_to_write_data(struct nfs_write_data *data, if (ret == PNFS_ATTEMPTED) nfs_inc_stats(inode, NFSIOS_PNFS_WRITE); else - _pnfs_clear_lseg_from_pages(&data->pages); + _pnfs_clear_lseg_from_pages(data->req, &data->pages); return ret; } @@ -153,7 +161,7 @@ pnfs_try_to_commit(struct nfs_write_data *data, if (ret == PNFS_ATTEMPTED) nfs_inc_stats(inode, NFSIOS_PNFS_COMMIT); else - _pnfs_clear_lseg_from_pages(&data->pages); + _pnfs_clear_lseg_from_pages(data->req, &data->pages); return ret; } -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 04/10] SQUASHME pnfs-submit: remove encode_layoutreturn 2010-06-23 19:21 ` [PATCH 03/10] SQUASHME pnfs-submit: clear page lseg on partial i/o andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 05/10] SQUASHME pnfs-submit: add error handling to layout return andros 0 siblings, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Not used by the file layout Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/nfs4xdr.c | 15 ++------------- include/linux/nfs4_pnfs.h | 4 ---- 2 files changed, 2 insertions(+), 17 deletions(-) diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c index a509bc1..fb50361 100644 --- a/fs/nfs/nfs4xdr.c +++ b/fs/nfs/nfs4xdr.c @@ -1874,24 +1874,13 @@ encode_layoutreturn(struct xdr_stream *xdr, *p++ = cpu_to_be32(args->lseg.iomode); *p = cpu_to_be32(args->return_type); if (args->return_type == RETURN_FILE) { - struct layoutdriver_io_operations *ld_io_ops = - NFS_SERVER(args->inode)->pnfs_curr_ld->ld_io_ops; - p = reserve_space(xdr, 16 + NFS4_STATEID_SIZE); p = xdr_encode_hyper(p, args->lseg.offset); p = xdr_encode_hyper(p, args->lseg.length); p = xdr_encode_opaque_fixed(p, &args->stateid.u.data, NFS4_STATEID_SIZE); - - dprintk("%s: call %pF\n", __func__, - ld_io_ops->encode_layoutreturn); - if (ld_io_ops->encode_layoutreturn) { - ld_io_ops->encode_layoutreturn( - &NFS_I(args->inode)->layout, xdr, args); - } else { - p = reserve_space(xdr, 4); - *p = cpu_to_be32(0); - } + p = reserve_space(xdr, 4); + *p = cpu_to_be32(0); } hdr->nops++; hdr->replen += decode_layoutreturn_maxsz; diff --git a/include/linux/nfs4_pnfs.h b/include/linux/nfs4_pnfs.h index d20b5de..782fdd9 100644 --- a/include/linux/nfs4_pnfs.h +++ b/include/linux/nfs4_pnfs.h @@ -154,10 +154,6 @@ struct layoutdriver_io_operations { struct pnfs_layout_segment * (*alloc_lseg) (struct pnfs_layout_type *layoutid, struct nfs4_pnfs_layoutget_res *lgr); void (*free_lseg) (struct pnfs_layout_segment *lseg); - void (*encode_layoutreturn) (struct pnfs_layout_type *layoutid, - struct xdr_stream *xdr, - const struct nfs4_pnfs_layoutreturn_arg *args); - /* Registration information for a new mounted file system */ int (*initialize_mountpoint) (struct nfs_client *); -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 05/10] SQUASHME pnfs-submit: add error handling to layout return 2010-06-23 19:21 ` [PATCH 04/10] SQUASHME pnfs-submit: remove encode_layoutreturn andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 06/10] SQUASHME pnfs-submit: handle assassinated layoutcommit andros 0 siblings, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/nfs4proc.c | 37 +++++++++++++++++++++++++++---------- 1 files changed, 27 insertions(+), 10 deletions(-) diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 5ffdfe1..0ab875f 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -5641,6 +5641,11 @@ static void nfs4_pnfs_layoutreturn_done(struct rpc_task *task, void *calldata) dprintk("--> %s\n", __func__); nfs4_sequence_done(server, &lrp->res.seq_res, task->tk_status); + if (RPC_ASSASSINATED(task)) + return; + + if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN) + nfs_restart_rpc(task, server->nfs_client); dprintk("<-- %s\n", __func__); } @@ -5667,7 +5672,7 @@ static const struct rpc_call_ops nfs4_pnfs_layoutreturn_call_ops = { .rpc_release = nfs4_pnfs_layoutreturn_release, }; -int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool wait) +int _pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool issync) { struct inode *ino = lrp->args.inode; struct nfs_server *server = NFS_SERVER(ino); @@ -5689,21 +5694,33 @@ int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool wait) dprintk("--> %s\n", __func__); lrp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE; task = rpc_run_task(&task_setup_data); - if (IS_ERR(task)) { - status = PTR_ERR(task); + if (IS_ERR(task)) + return PTR_ERR(task); + if (!issync) goto out; - } - if (wait) { - status = nfs4_wait_for_completion_rpc_task(task); - if (status == 0) - status = task->tk_status; - } - rpc_put_task(task); + status = nfs4_wait_for_completion_rpc_task(task); + if (status != 0) + goto out; + status = task->tk_status; out: dprintk("<-- %s\n", __func__); + rpc_put_task(task); return status; } +int pnfs4_proc_layoutreturn(struct nfs4_pnfs_layoutreturn *lrp, bool issync) +{ + struct nfs_server *server = NFS_SERVER(lrp->args.inode); + struct nfs4_exception exception = { }; + int err; + do { + err = nfs4_handle_exception(server, + _pnfs4_proc_layoutreturn(lrp, issync), + &exception); + } while (exception.retry); + return err; +} + int nfs4_pnfs_getdeviceinfo(struct nfs_server *server, struct pnfs_device *pdev) { struct nfs4_pnfs_getdeviceinfo_arg args = { -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 06/10] SQUASHME pnfs-submit: handle assassinated layoutcommit 2010-06-23 19:21 ` [PATCH 05/10] SQUASHME pnfs-submit: add error handling to layout return andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get andros 0 siblings, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/nfs4proc.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 0ab875f..35e8a56 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -5539,6 +5539,8 @@ pnfs_layoutcommit_done(struct rpc_task *task, void *calldata) data->status = task->tk_status; nfs4_sequence_done(server, &data->res.seq_res, task->tk_status); + if (RPC_ASSASSINATED(task)) + return; if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN) nfs_restart_rpc(task, server->nfs_client); -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get 2010-06-23 19:21 ` [PATCH 06/10] SQUASHME pnfs-submit: handle assassinated layoutcommit andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 08/10] pnfs-post-submit: restore encode_layoutreturn andros 2010-06-30 15:19 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get Boaz Harrosh 0 siblings, 2 replies; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Redo pnfs4_proc_layoutget() logic to match the other async operations that call nfs4_wait_for_completion_rpc_task. pnfs_get_layout_done passes session and delay errors. Handle them with the generic error handlers. Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/nfs4proc.c | 45 +++++++++++++++++++++++++++++---------------- 1 files changed, 29 insertions(+), 16 deletions(-) diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 35e8a56..6283996 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -5446,6 +5446,10 @@ static void nfs4_pnfs_layoutget_done(struct rpc_task *task, void *calldata) return; pnfs_get_layout_done(lgp, task->tk_status); + + if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN) + nfs_restart_rpc(task, server->nfs_client); + dprintk("<-- %s\n", __func__); } @@ -5471,10 +5475,9 @@ static const struct rpc_call_ops nfs4_pnfs_layoutget_call_ops = { * and deal with retries. * Currently we can't since we release lgp and its contents. */ -int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) +static int _pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) { - struct inode *ino = lgp->args.inode; - struct nfs_server *server = NFS_SERVER(ino); + struct nfs_server *server = NFS_SERVER(lgp->args.inode); struct rpc_task *task; struct rpc_message msg = { .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_PNFS_LAYOUTGET], @@ -5488,35 +5491,45 @@ int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) .callback_data = lgp, .flags = RPC_TASK_ASYNC, }; - int status; + int status = 0; dprintk("--> %s\n", __func__); lgp->res.layout.buf = (void *)__get_free_page(GFP_NOFS); if (lgp->res.layout.buf == NULL) { nfs4_pnfs_layoutget_release(lgp); - status = -ENOMEM; - goto out; + return -ENOMEM; } lgp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE; task = rpc_run_task(&task_setup_data); - if (IS_ERR(task)) { - status = PTR_ERR(task); - goto out; - } + if (IS_ERR(task)) + return PTR_ERR(task); status = nfs4_wait_for_completion_rpc_task(task); - if (status == 0) { - status = lgp->status; - if (status == 0) - status = pnfs_layout_process(lgp); - } - rpc_put_task(task); + if (status != 0) + goto out; + status = lgp->status; + if (status != 0) + goto out; + status = pnfs_layout_process(lgp); out: + rpc_put_task(task); dprintk("<-- %s status=%d\n", __func__, status); return status; } +int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) +{ + struct nfs_server *server = NFS_SERVER(lgp->args.inode); + struct nfs4_exception exception = { }; + int err; + do { + err = nfs4_handle_exception(server, _pnfs4_proc_layoutget(lgp), + &exception); + } while (exception.retry); + return err; +} + static void pnfs_layoutcommit_prepare(struct rpc_task *task, void *data) { struct pnfs_layoutcommit_data *ldata = -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 08/10] pnfs-post-submit: restore encode_layoutreturn 2010-06-23 19:21 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 09/10] SQUASHME: pnfs-submit: don't re-initialize i_lock andros 2010-06-30 15:19 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get Boaz Harrosh 1 sibling, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/nfs4xdr.c | 14 ++++++++++++-- include/linux/nfs4_pnfs.h | 4 ++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c index fb50361..eeee855 100644 --- a/fs/nfs/nfs4xdr.c +++ b/fs/nfs/nfs4xdr.c @@ -1874,13 +1874,23 @@ encode_layoutreturn(struct xdr_stream *xdr, *p++ = cpu_to_be32(args->lseg.iomode); *p = cpu_to_be32(args->return_type); if (args->return_type == RETURN_FILE) { + struct layoutdriver_io_operations *ld_io_ops = + NFS_SERVER(args->inode)->pnfs_curr_ld->ld_io_ops; + p = reserve_space(xdr, 16 + NFS4_STATEID_SIZE); p = xdr_encode_hyper(p, args->lseg.offset); p = xdr_encode_hyper(p, args->lseg.length); p = xdr_encode_opaque_fixed(p, &args->stateid.u.data, NFS4_STATEID_SIZE); - p = reserve_space(xdr, 4); - *p = cpu_to_be32(0); + dprintk("%s: call %pF\n", __func__, + ld_io_ops->encode_layoutreturn); + if (ld_io_ops->encode_layoutreturn) { + ld_io_ops->encode_layoutreturn( + &NFS_I(args->inode)->layout, xdr, args); + } else { + p = reserve_space(xdr, 4); + *p = cpu_to_be32(0); + } } hdr->nops++; hdr->replen += decode_layoutreturn_maxsz; diff --git a/include/linux/nfs4_pnfs.h b/include/linux/nfs4_pnfs.h index 782fdd9..d20b5de 100644 --- a/include/linux/nfs4_pnfs.h +++ b/include/linux/nfs4_pnfs.h @@ -154,6 +154,10 @@ struct layoutdriver_io_operations { struct pnfs_layout_segment * (*alloc_lseg) (struct pnfs_layout_type *layoutid, struct nfs4_pnfs_layoutget_res *lgr); void (*free_lseg) (struct pnfs_layout_segment *lseg); + void (*encode_layoutreturn) (struct pnfs_layout_type *layoutid, + struct xdr_stream *xdr, + const struct nfs4_pnfs_layoutreturn_arg *args); + /* Registration information for a new mounted file system */ int (*initialize_mountpoint) (struct nfs_client *); -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 09/10] SQUASHME: pnfs-submit: don't re-initialize i_lock 2010-06-23 19:21 ` [PATCH 08/10] pnfs-post-submit: restore encode_layoutreturn andros @ 2010-06-23 19:21 ` andros 2010-06-23 19:21 ` [PATCH 10/10] SQUASHME pnfs-submit: remove struct nfs_server from stack andros 0 siblings, 1 reply; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Initialized in the VFS (inode_init_always) Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/inode.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index d6dd6e5..7989dea 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -1420,7 +1420,6 @@ static void pnfs_init_once(struct nfs_inode *nfsi) { #ifdef CONFIG_NFS_V4_1 init_waitqueue_head(&nfsi->lo_waitq); - spin_lock_init(&nfsi->vfs_inode.i_lock); seqlock_init(&nfsi->layout.seqlock); INIT_LIST_HEAD(&nfsi->layout.lo_layouts); INIT_LIST_HEAD(&nfsi->layout.segs); -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 10/10] SQUASHME pnfs-submit: remove struct nfs_server from stack 2010-06-23 19:21 ` [PATCH 09/10] SQUASHME: pnfs-submit: don't re-initialize i_lock andros @ 2010-06-23 19:21 ` andros 0 siblings, 0 replies; 20+ messages in thread From: andros @ 2010-06-23 19:21 UTC (permalink / raw) To: bhalevy; +Cc: linux-nfs, Andy Adamson From: Andy Adamson <andros@netapp.com> Signed-off-by: Andy Adamson <andros@netapp.com> --- fs/nfs/nfs4filelayoutdev.c | 18 ++++++++++++------ 1 files changed, 12 insertions(+), 6 deletions(-) diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c index 5842c09..8e9d04a 100644 --- a/fs/nfs/nfs4filelayoutdev.c +++ b/fs/nfs/nfs4filelayoutdev.c @@ -121,9 +121,7 @@ _data_server_lookup(u32 ip_addr, u32 port) static int nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds) { - struct nfs_server tmp = { - .nfs_client = NULL, - }; + struct nfs_server *tmp; struct sockaddr_in sin; struct rpc_clnt *mds_clnt = mds_srv->client; struct nfs_client *clp = mds_srv->nfs_client; @@ -154,12 +152,18 @@ nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds) } goto out; } + + /* Temporay server for nfs4_set_client */ + tmp = kzalloc(sizeof(struct nfs_server), GFP_KERNEL); + if (!tmp) + goto out; + /* * Set a retrans, timeout interval, and authflavor equual to the MDS * values. Use the MDS nfs_client cl_ipaddr field so as to use the * same co_ownerid as the MDS. */ - err = nfs4_set_client(&tmp, + err = nfs4_set_client(tmp, mds_srv->nfs_client->cl_hostname, (struct sockaddr *)&sin, sizeof(struct sockaddr), @@ -169,9 +173,9 @@ nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds) mds_clnt->cl_xprt->timeout, 1 /* minorversion */); if (err < 0) - goto out; + goto out_free; - clp = tmp.nfs_client; + clp = tmp->nfs_client; /* Ask for only the EXCHGID4_FLAG_USE_PNFS_DS pNFS role */ dprintk("%s EXCHANGE_ID for clp %p\n", __func__, clp); @@ -211,6 +215,8 @@ nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds) dprintk("%s: ip=%x, port=%hu, rpcclient %p\n", __func__, ntohl(ds->ds_ip_addr), ntohs(ds->ds_port), clp->cl_rpcclient); +out_free: + kfree(tmp); out: dprintk("%s Returns %d\n", __func__, err); return err; -- 1.6.6 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get 2010-06-23 19:21 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get andros 2010-06-23 19:21 ` [PATCH 08/10] pnfs-post-submit: restore encode_layoutreturn andros @ 2010-06-30 15:19 ` Boaz Harrosh 2010-06-30 19:23 ` William A. (Andy) Adamson 1 sibling, 1 reply; 20+ messages in thread From: Boaz Harrosh @ 2010-06-30 15:19 UTC (permalink / raw) To: andros; +Cc: bhalevy, linux-nfs On 06/23/2010 10:21 PM, andros@netapp.com wrote: > From: Andy Adamson <andros@netapp.com> > > Redo pnfs4_proc_layoutget() logic to match the other async operations that call > nfs4_wait_for_completion_rpc_task. > > pnfs_get_layout_done passes session and delay errors. Handle them with the > generic error handlers. > > Signed-off-by: Andy Adamson <andros@netapp.com> > --- > fs/nfs/nfs4proc.c | 45 +++++++++++++++++++++++++++++---------------- > 1 files changed, 29 insertions(+), 16 deletions(-) > > diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c > index 35e8a56..6283996 100644 > --- a/fs/nfs/nfs4proc.c > +++ b/fs/nfs/nfs4proc.c > @@ -5446,6 +5446,10 @@ static void nfs4_pnfs_layoutget_done(struct rpc_task *task, void *calldata) > return; > > pnfs_get_layout_done(lgp, task->tk_status); > + > + if (nfs4_async_handle_error(task, server, NULL, NULL) == -EAGAIN) > + nfs_restart_rpc(task, server->nfs_client); > + Andy high. In pnfs_get_layout_done() it is working hard to do "lgp->status = XXX" I never understood, what gets to look at that later on? Thanks Boaz > dprintk("<-- %s\n", __func__); > } > > @@ -5471,10 +5475,9 @@ static const struct rpc_call_ops nfs4_pnfs_layoutget_call_ops = { > * and deal with retries. > * Currently we can't since we release lgp and its contents. > */ > -int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) > +static int _pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) > { > - struct inode *ino = lgp->args.inode; > - struct nfs_server *server = NFS_SERVER(ino); > + struct nfs_server *server = NFS_SERVER(lgp->args.inode); > struct rpc_task *task; > struct rpc_message msg = { > .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_PNFS_LAYOUTGET], > @@ -5488,35 +5491,45 @@ int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) > .callback_data = lgp, > .flags = RPC_TASK_ASYNC, > }; > - int status; > + int status = 0; > > dprintk("--> %s\n", __func__); > > lgp->res.layout.buf = (void *)__get_free_page(GFP_NOFS); > if (lgp->res.layout.buf == NULL) { > nfs4_pnfs_layoutget_release(lgp); > - status = -ENOMEM; > - goto out; > + return -ENOMEM; > } > > lgp->res.seq_res.sr_slotid = NFS4_MAX_SLOT_TABLE; > task = rpc_run_task(&task_setup_data); > - if (IS_ERR(task)) { > - status = PTR_ERR(task); > - goto out; > - } > + if (IS_ERR(task)) > + return PTR_ERR(task); > status = nfs4_wait_for_completion_rpc_task(task); > - if (status == 0) { > - status = lgp->status; > - if (status == 0) > - status = pnfs_layout_process(lgp); > - } > - rpc_put_task(task); > + if (status != 0) > + goto out; > + status = lgp->status; > + if (status != 0) > + goto out; > + status = pnfs_layout_process(lgp); > out: > + rpc_put_task(task); > dprintk("<-- %s status=%d\n", __func__, status); > return status; > } > > +int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) > +{ > + struct nfs_server *server = NFS_SERVER(lgp->args.inode); > + struct nfs4_exception exception = { }; > + int err; > + do { > + err = nfs4_handle_exception(server, _pnfs4_proc_layoutget(lgp), > + &exception); > + } while (exception.retry); > + return err; > +} > + > static void pnfs_layoutcommit_prepare(struct rpc_task *task, void *data) > { > struct pnfs_layoutcommit_data *ldata = ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get 2010-06-30 15:19 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get Boaz Harrosh @ 2010-06-30 19:23 ` William A. (Andy) Adamson 0 siblings, 0 replies; 20+ messages in thread From: William A. (Andy) Adamson @ 2010-06-30 19:23 UTC (permalink / raw) To: Boaz Harrosh; +Cc: bhalevy, linux-nfs On Wed, Jun 30, 2010 at 11:19 AM, Boaz Harrosh <bharrosh@panasas.com> w= rote: > On 06/23/2010 10:21 PM, andros@netapp.com wrote: >> From: Andy Adamson <andros@netapp.com> >> >> Redo pnfs4_proc_layoutget() logic to match the other async operation= s that call >> nfs4_wait_for_completion_rpc_task. >> >> pnfs_get_layout_done passes session and delay errors. Handle them wi= th the >> generic error handlers. >> >> Signed-off-by: Andy Adamson <andros@netapp.com> >> --- >> =A0fs/nfs/nfs4proc.c | =A0 45 +++++++++++++++++++++++++++++---------= ------- >> =A01 files changed, 29 insertions(+), 16 deletions(-) >> >> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c >> index 35e8a56..6283996 100644 >> --- a/fs/nfs/nfs4proc.c >> +++ b/fs/nfs/nfs4proc.c >> @@ -5446,6 +5446,10 @@ static void nfs4_pnfs_layoutget_done(struct r= pc_task *task, void *calldata) >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 return; >> >> =A0 =A0 =A0 pnfs_get_layout_done(lgp, task->tk_status); >> + >> + =A0 =A0 if (nfs4_async_handle_error(task, server, NULL, NULL) =3D=3D= -EAGAIN) >> + =A0 =A0 =A0 =A0 =A0 =A0 nfs_restart_rpc(task, server->nfs_client); >> + > > Andy high. > > In pnfs_get_layout_done() it is working hard to do "lgp->status =3D X= XX" > > I never understood, what gets to look at that later on? HI Boaz _pnfs4_proc_layoutget will return it if it is set, then send_layoutget returns it to pnfs_update_layout. -->Andy > > Thanks > Boaz > >> =A0 =A0 =A0 dprintk("<-- %s\n", __func__); >> =A0} >> >> @@ -5471,10 +5475,9 @@ static const struct rpc_call_ops nfs4_pnfs_la= youtget_call_ops =3D { >> =A0 * and deal with retries. >> =A0 * Currently we can't since we release lgp and its contents. >> =A0 */ >> -int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) >> +static int _pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) >> =A0{ >> - =A0 =A0 struct inode *ino =3D lgp->args.inode; >> - =A0 =A0 struct nfs_server *server =3D NFS_SERVER(ino); >> + =A0 =A0 struct nfs_server *server =3D NFS_SERVER(lgp->args.inode); >> =A0 =A0 =A0 struct rpc_task *task; >> =A0 =A0 =A0 struct rpc_message msg =3D { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 .rpc_proc =3D &nfs4_procedures[NFSPROC4_= CLNT_PNFS_LAYOUTGET], >> @@ -5488,35 +5491,45 @@ int pnfs4_proc_layoutget(struct nfs4_pnfs_la= youtget *lgp) >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 .callback_data =3D lgp, >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 .flags =3D RPC_TASK_ASYNC, >> =A0 =A0 =A0 }; >> - =A0 =A0 int status; >> + =A0 =A0 int status =3D 0; >> >> =A0 =A0 =A0 dprintk("--> %s\n", __func__); >> >> =A0 =A0 =A0 lgp->res.layout.buf =3D (void *)__get_free_page(GFP_NOFS= ); >> =A0 =A0 =A0 if (lgp->res.layout.buf =3D=3D NULL) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 nfs4_pnfs_layoutget_release(lgp); >> - =A0 =A0 =A0 =A0 =A0 =A0 status =3D -ENOMEM; >> - =A0 =A0 =A0 =A0 =A0 =A0 goto out; >> + =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM; >> =A0 =A0 =A0 } >> >> =A0 =A0 =A0 lgp->res.seq_res.sr_slotid =3D NFS4_MAX_SLOT_TABLE; >> =A0 =A0 =A0 task =3D rpc_run_task(&task_setup_data); >> - =A0 =A0 if (IS_ERR(task)) { >> - =A0 =A0 =A0 =A0 =A0 =A0 status =3D PTR_ERR(task); >> - =A0 =A0 =A0 =A0 =A0 =A0 goto out; >> - =A0 =A0 } >> + =A0 =A0 if (IS_ERR(task)) >> + =A0 =A0 =A0 =A0 =A0 =A0 return PTR_ERR(task); >> =A0 =A0 =A0 status =3D nfs4_wait_for_completion_rpc_task(task); >> - =A0 =A0 if (status =3D=3D 0) { >> - =A0 =A0 =A0 =A0 =A0 =A0 status =3D lgp->status; >> - =A0 =A0 =A0 =A0 =A0 =A0 if (status =3D=3D 0) >> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 status =3D pnfs_layout_pro= cess(lgp); >> - =A0 =A0 } >> - =A0 =A0 rpc_put_task(task); >> + =A0 =A0 if (status !=3D 0) >> + =A0 =A0 =A0 =A0 =A0 =A0 goto out; >> + =A0 =A0 status =3D lgp->status; >> + =A0 =A0 if (status !=3D 0) >> + =A0 =A0 =A0 =A0 =A0 =A0 goto out; >> + =A0 =A0 status =3D pnfs_layout_process(lgp); >> =A0out: >> + =A0 =A0 rpc_put_task(task); >> =A0 =A0 =A0 dprintk("<-- %s status=3D%d\n", __func__, status); >> =A0 =A0 =A0 return status; >> =A0} >> >> +int pnfs4_proc_layoutget(struct nfs4_pnfs_layoutget *lgp) >> +{ >> + =A0 =A0 struct nfs_server *server =3D NFS_SERVER(lgp->args.inode); >> + =A0 =A0 struct nfs4_exception exception =3D { }; >> + =A0 =A0 int err; >> + =A0 =A0 do { >> + =A0 =A0 =A0 =A0 =A0 =A0 err =3D nfs4_handle_exception(server, _pnf= s4_proc_layoutget(lgp), >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 &exception); >> + =A0 =A0 } while (exception.retry); >> + =A0 =A0 return err; >> +} >> + >> =A0static void pnfs_layoutcommit_prepare(struct rpc_task *task, void= *data) >> =A0{ >> =A0 =A0 =A0 struct pnfs_layoutcommit_data *ldata =3D > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" = in > the body of a message to majordomo@vger.kernel.org > More majordomo info at =A0http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 2010-06-23 19:21 [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 andros 2010-06-23 19:21 ` [PATCH 01/10] SQUASHME: pnfs-submit remove unused filelayout_mount_type andros @ 2010-06-24 13:14 ` Benny Halevy 2010-06-24 17:02 ` William A. (Andy) Adamson 2010-07-01 18:27 ` Benny Halevy 2 siblings, 1 reply; 20+ messages in thread From: Benny Halevy @ 2010-06-24 13:14 UTC (permalink / raw) To: andros; +Cc: linux-nfs On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote: > Responded to comments, added a 2 cleanup patchses > > Plus some code cleanup > 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch > > and some bug fixes > 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch > > NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch > was replaced by: > 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch > > > Remove unused (by file layout) encode_layoutreturn io operation > 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch > 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch > > 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch > > Note: pnfs4_proc_layoutget is only called by send_layout() which prints > the status. > 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch > > Add back encode_layoutreturn io operation > 0008-pnfs-post-submit-restore-encode_layoutreturn.patch > > > New patches: > 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch > > This gets rid of a frame stack warning; > 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch > > Testing: > --------- > > CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS > Passes Connectathon tests > > Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the > pyNFS server and the testclient framework. > > Still todo: > > Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and > layoutcommit do not pass nfs_stste to the error handlers. > > Handle NFS4ERR_BAD_LAYOUT. > > CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. > > -->Andy Andy, I've hit BUG_ON(lo->refcount <= 0); in put_layout() with this patchset. I'm not sure if it introduced it or not, still investigating... Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: layout.refcount 1 Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------ Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341! Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/initstate Jun 24 12:07:26 tl2 kernel: CPU 1 Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore snd_page_alloc button video output rng_core sg cdrom ata_generic ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode] Jun 24 12:07:26 tl2 kernel: Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527 Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>] [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 EFLAGS: 00010246 Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: ffff8800704b6b78 RCX: 0000000000000066 Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: ffffea0001b931a8 RDI: ffff8800704b6b78 Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: 0000000000000000 R09: ffff88007356a500 Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: 0000000000000003 R12: ffff8800704b69a8 Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: ffff88007356a508 R15: ffff88007356a590 Jun 24 12:07:26 tl2 kernel: FS: 0000000000000000(0000) GS:ffff880001a80000(0000) knlGS:0000000000000000 Jun 24 12:07:26 tl2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: 0000000001698000 CR4: 00000000000406e0 Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo ffff88007525c000, task ffff88007d988000) Jun 24 12:07:26 tl2 kernel: Stack: Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 ffff88007525dd60 ffffffffa05d203f Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 ffff880073854f00 ffffffffa05d5880 Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c ffff88007525dd90 ffff88007356a500 Jun 24 12:07:26 tl2 kernel: Call Trace: Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release+0x43/0x68 [nfs] Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs] Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] rpc_release_calldata+0x17/0x19 [sunrpc] Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/0x66 [sunrpc] Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task+0x98/0x9c [sunrpc] Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute+0x205/0x212 [sunrpc] Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule+0x15/0x17 [sunrpc] Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread+0x1aa/0x23b Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? rpc_async_schedule+0x0/0x17 [sunrpc] Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? autoremove_wake_function+0x0/0x39 Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? spin_unlock_irqrestore+0xe/0x10 Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread+0x0/0x23b Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87 Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] kernel_thread_helper+0x4/0x10 Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87 Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? kernel_thread_helper+0x0/0x10 Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67 48 8b 82 48 03 00 00 f6 05 Jun 24 12:07:26 tl2 kernel: RIP [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]--- ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 2010-06-24 13:14 ` [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 Benny Halevy @ 2010-06-24 17:02 ` William A. (Andy) Adamson [not found] ` <AANLkTikJWftkWhU8TIOGxvGxo8s2_sXyMn8VIsk9caTv-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: William A. (Andy) Adamson @ 2010-06-24 17:02 UTC (permalink / raw) To: Benny Halevy; +Cc: linux-nfs OK - I'll look into it. Sorry I missed today's pNFS call. -->Andy On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com> wrote: > On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote: >> Responded to comments, added a 2 cleanup patchses >> >> Plus some code cleanup >> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch >> >> and some bug fixes >> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch >> >> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch >> was replaced by: >> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch >> >> >> Remove unused (by file layout) encode_layoutreturn io operation >> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch >> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch >> >> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch >> >> Note: pnfs4_proc_layoutget is only called by send_layout() which prints >> the status. >> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch >> >> Add back encode_layoutreturn io operation >> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch >> >> >> New patches: >> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch >> >> This gets rid of a frame stack warning; >> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch >> >> Testing: >> --------- >> >> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS >> Passes Connectathon tests >> >> Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the >> pyNFS server and the testclient framework. >> >> Still todo: >> >> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and >> layoutcommit do not pass nfs_stste to the error handlers. >> >> Handle NFS4ERR_BAD_LAYOUT. >> >> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. >> >> -->Andy > > Andy, I've hit > BUG_ON(lo->refcount <= 0); > in put_layout() with this patchset. > I'm not sure if it introduced it or not, still investigating... > > Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: layout.refcount 1 > Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------ > Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341! > Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC > Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/initstate > Jun 24 12:07:26 tl2 kernel: CPU 1 > Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore snd_page_alloc button video output rng_core sg cdrom ata_generic ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode] > Jun 24 12:07:26 tl2 kernel: > Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527 > Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>] [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] > Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 EFLAGS: 00010246 > Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: ffff8800704b6b78 RCX: 0000000000000066 > Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: ffffea0001b931a8 RDI: ffff8800704b6b78 > Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: 0000000000000000 R09: ffff88007356a500 > Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: 0000000000000003 R12: ffff8800704b69a8 > Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: ffff88007356a508 R15: ffff88007356a590 > Jun 24 12:07:26 tl2 kernel: FS: 0000000000000000(0000) GS:ffff880001a80000(0000) knlGS:0000000000000000 > Jun 24 12:07:26 tl2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: 0000000001698000 CR4: 00000000000406e0 > Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo ffff88007525c000, task ffff88007d988000) > Jun 24 12:07:26 tl2 kernel: Stack: > Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 ffff88007525dd60 ffffffffa05d203f > Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 ffff880073854f00 ffffffffa05d5880 > Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c ffff88007525dd90 ffff88007356a500 > Jun 24 12:07:26 tl2 kernel: Call Trace: > Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release+0x43/0x68 [nfs] > Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs] > Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] rpc_release_calldata+0x17/0x19 [sunrpc] > Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/0x66 [sunrpc] > Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task+0x98/0x9c [sunrpc] > Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute+0x205/0x212 [sunrpc] > Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule+0x15/0x17 [sunrpc] > Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread+0x1aa/0x23b > Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? rpc_async_schedule+0x0/0x17 [sunrpc] > Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? autoremove_wake_function+0x0/0x39 > Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? spin_unlock_irqrestore+0xe/0x10 > Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread+0x0/0x23b > Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87 > Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] kernel_thread_helper+0x4/0x10 > Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87 > Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? kernel_thread_helper+0x0/0x10 > Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67 48 8b 82 48 03 00 00 f6 05 > Jun 24 12:07:26 tl2 kernel: RIP [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] > Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20> > Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]--- > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <AANLkTikJWftkWhU8TIOGxvGxo8s2_sXyMn8VIsk9caTv-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 [not found] ` <AANLkTikJWftkWhU8TIOGxvGxo8s2_sXyMn8VIsk9caTv-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2010-06-28 16:44 ` Andy Adamson 2010-06-28 18:53 ` Benny Halevy 0 siblings, 1 reply; 20+ messages in thread From: Andy Adamson @ 2010-06-28 16:44 UTC (permalink / raw) To: Benny Halevy; +Cc: linux-nfs@vger.kernel.org Mailing list Hi Benny I have not been able to reproduce this BUG. I've tried against the files pyNFS server with return_on_close False as well a True, and against a GFS2/pNFS cluster with write layouts turned on. Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch calls put_lseg when I/O to a DS fails. I tested this using the pyNFS files layout server and blocking the DS with iptables. I think this is the only change in this patch set that would affect the refcounting. Are you able to reproduce the BUG? -->Andy On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote: > OK - I'll look into it. > > Sorry I missed today's pNFS call. > > -->Andy > > On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com> > wrote: >> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote: >>> Responded to comments, added a 2 cleanup patchses >>> >>> Plus some code cleanup >>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch >>> >>> and some bug fixes >>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch >>> >>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the- >>> MDS.patch >>> was replaced by: >>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch >>> >>> >>> Remove unused (by file layout) encode_layoutreturn io operation >>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch >>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch >>> >>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch >>> >>> Note: pnfs4_proc_layoutget is only called by send_layout() which >>> prints >>> the status. >>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch >>> >>> Add back encode_layoutreturn io operation >>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch >>> >>> >>> New patches: >>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch >>> >>> This gets rid of a frame stack warning; >>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch >>> >>> Testing: >>> --------- >>> >>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS >>> Passes Connectathon tests >>> >>> Tested layoutget and layoutreturn recovery from >>> NFS4ERR_DEAD_SESSION with the >>> pyNFS server and the testclient framework. >>> >>> Still todo: >>> >>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, >>> layoutget, and >>> layoutcommit do not pass nfs_stste to the error handlers. >>> >>> Handle NFS4ERR_BAD_LAYOUT. >>> >>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. >>> >>> -->Andy >> >> Andy, I've hit >> BUG_ON(lo->refcount <= 0); >> in put_layout() with this patchset. >> I'm not sure if it introduced it or not, still investigating... >> >> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: >> layout.refcount 1 >> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------ >> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/ >> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341! >> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP >> DEBUG_PAGEALLOC >> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/ >> initstate >> Jun 24 12:07:26 tl2 kernel: CPU 1 >> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd >> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 >> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi >> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror >> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm >> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel >> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core >> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore >> snd_page_alloc button video output rng_core sg cdrom ata_generic >> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd >> ohci_hcd ehci_hcd [last unloaded: microcode] >> Jun 24 12:07:26 tl2 kernel: >> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted >> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527 >> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>] >> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] >> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 EFLAGS: >> 00010246 >> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: >> ffff8800704b6b78 RCX: 0000000000000066 >> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: >> ffffea0001b931a8 RDI: ffff8800704b6b78 >> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: >> 0000000000000000 R09: ffff88007356a500 >> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: >> 0000000000000003 R12: ffff8800704b69a8 >> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: >> ffff88007356a508 R15: ffff88007356a590 >> Jun 24 12:07:26 tl2 kernel: FS: 0000000000000000(0000) >> GS:ffff880001a80000(0000) knlGS:0000000000000000 >> Jun 24 12:07:26 tl2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: >> 000000008005003b >> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: >> 0000000001698000 CR4: 00000000000406e0 >> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: >> 0000000000000000 DR2: 0000000000000000 >> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: >> 00000000ffff0ff0 DR7: 0000000000000400 >> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo >> ffff88007525c000, task ffff88007d988000) >> Jun 24 12:07:26 tl2 kernel: Stack: >> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 >> ffff88007525dd60 ffffffffa05d203f >> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 >> ffff880073854f00 ffffffffa05d5880 >> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c >> ffff88007525dd90 ffff88007356a500 >> Jun 24 12:07:26 tl2 kernel: Call Trace: >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release >> +0x43/0x68 [nfs] >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] >> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs] >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] >> rpc_release_calldata+0x17/0x19 [sunrpc] >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/ >> 0x66 [sunrpc] >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task >> +0x98/0x9c [sunrpc] >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute >> +0x205/0x212 [sunrpc] >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule >> +0x15/0x17 [sunrpc] >> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread >> +0x1aa/0x23b >> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? >> rpc_async_schedule+0x0/0x17 [sunrpc] >> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? >> autoremove_wake_function+0x0/0x39 >> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? >> spin_unlock_irqrestore+0xe/0x10 >> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread >> +0x0/0x23b >> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87 >> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] >> kernel_thread_helper+0x4/0x10 >> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87 >> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? >> kernel_thread_helper+0x0/0x10 >> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24 >> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 >> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67 >> 48 8b 82 48 03 00 00 f6 05 >> Jun 24 12:07:26 tl2 kernel: RIP [<ffffffffa05d0ea4>] put_layout >> +0x2f/0xa7 [nfs] >> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20> >> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]--- >> -- >> To unsubscribe from this list: send the line "unsubscribe linux- >> nfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" > in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 2010-06-28 16:44 ` Andy Adamson @ 2010-06-28 18:53 ` Benny Halevy 2010-06-28 19:22 ` William A. (Andy) Adamson 0 siblings, 1 reply; 20+ messages in thread From: Benny Halevy @ 2010-06-28 18:53 UTC (permalink / raw) To: Andy Adamson; +Cc: linux-nfs@vger.kernel.org Mailing list On Jun. 28, 2010, 19:44 +0300, Andy Adamson <andros@netapp.com> wrote: > Hi Benny > > I have not been able to reproduce this BUG. I've tried against the > files pyNFS server with return_on_close False as well a True, and > against a GFS2/pNFS cluster with write layouts turned on. > > Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch > calls put_lseg when I/O to a DS fails. I tested this using the pyNFS > files layout server and blocking the DS with iptables. I think this is > the only change in this patch set that would affect the refcounting. > > Are you able to reproduce the BUG? The easiest way I found to reproduce this bug is running the cthon tests on a locally mounted file system exported over PNFSD_LOCAL_EXPORT. The test machine is a dual core SMP machine. Are you testing over a VM? Is it uni-processor? Benny > > -->Andy > > On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote: > >> OK - I'll look into it. >> >> Sorry I missed today's pNFS call. >> >> -->Andy >> >> On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com> >> wrote: >>> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote: >>>> Responded to comments, added a 2 cleanup patchses >>>> >>>> Plus some code cleanup >>>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch >>>> >>>> and some bug fixes >>>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch >>>> >>>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the- >>>> MDS.patch >>>> was replaced by: >>>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch >>>> >>>> >>>> Remove unused (by file layout) encode_layoutreturn io operation >>>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch >>>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch >>>> >>>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch >>>> >>>> Note: pnfs4_proc_layoutget is only called by send_layout() which >>>> prints >>>> the status. >>>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch >>>> >>>> Add back encode_layoutreturn io operation >>>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch >>>> >>>> >>>> New patches: >>>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch >>>> >>>> This gets rid of a frame stack warning; >>>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch >>>> >>>> Testing: >>>> --------- >>>> >>>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS >>>> Passes Connectathon tests >>>> >>>> Tested layoutget and layoutreturn recovery from >>>> NFS4ERR_DEAD_SESSION with the >>>> pyNFS server and the testclient framework. >>>> >>>> Still todo: >>>> >>>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, >>>> layoutget, and >>>> layoutcommit do not pass nfs_stste to the error handlers. >>>> >>>> Handle NFS4ERR_BAD_LAYOUT. >>>> >>>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. >>>> >>>> -->Andy >>> >>> Andy, I've hit >>> BUG_ON(lo->refcount <= 0); >>> in put_layout() with this patchset. >>> I'm not sure if it introduced it or not, still investigating... >>> >>> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: >>> layout.refcount 1 >>> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------ >>> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy/ >>> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341! >>> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP >>> DEBUG_PAGEALLOC >>> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/ >>> initstate >>> Jun 24 12:07:26 tl2 kernel: CPU 1 >>> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfsd >>> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 >>> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi >>> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror >>> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm >>> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel >>> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core >>> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore >>> snd_page_alloc button video output rng_core sg cdrom ata_generic >>> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd >>> ohci_hcd ehci_hcd [last unloaded: microcode] >>> Jun 24 12:07:26 tl2 kernel: >>> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted >>> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527 >>> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>] >>> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] >>> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 EFLAGS: >>> 00010246 >>> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: >>> ffff8800704b6b78 RCX: 0000000000000066 >>> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: >>> ffffea0001b931a8 RDI: ffff8800704b6b78 >>> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: >>> 0000000000000000 R09: ffff88007356a500 >>> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: >>> 0000000000000003 R12: ffff8800704b69a8 >>> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: >>> ffff88007356a508 R15: ffff88007356a590 >>> Jun 24 12:07:26 tl2 kernel: FS: 0000000000000000(0000) >>> GS:ffff880001a80000(0000) knlGS:0000000000000000 >>> Jun 24 12:07:26 tl2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: >>> 000000008005003b >>> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: >>> 0000000001698000 CR4: 00000000000406e0 >>> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: >>> 0000000000000000 DR2: 0000000000000000 >>> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: >>> 00000000ffff0ff0 DR7: 0000000000000400 >>> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinfo >>> ffff88007525c000, task ffff88007d988000) >>> Jun 24 12:07:26 tl2 kernel: Stack: >>> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 >>> ffff88007525dd60 ffffffffa05d203f >>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 >>> ffff880073854f00 ffffffffa05d5880 >>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c >>> ffff88007525dd90 ffff88007356a500 >>> Jun 24 12:07:26 tl2 kernel: Call Trace: >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_release >>> +0x43/0x68 [nfs] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] >>> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] >>> rpc_release_calldata+0x17/0x19 [sunrpc] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5e/ >>> 0x66 [sunrpc] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task >>> +0x98/0x9c [sunrpc] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute >>> +0x205/0x212 [sunrpc] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedule >>> +0x15/0x17 [sunrpc] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread >>> +0x1aa/0x23b >>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? >>> rpc_async_schedule+0x0/0x17 [sunrpc] >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? >>> autoremove_wake_function+0x0/0x39 >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? >>> spin_unlock_irqrestore+0xe/0x10 >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread >>> +0x0/0x23b >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87 >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] >>> kernel_thread_helper+0x4/0x10 >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x87 >>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? >>> kernel_thread_helper+0x0/0x10 >>> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24 >>> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 >>> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 67 >>> 48 8b 82 48 03 00 00 f6 05 >>> Jun 24 12:07:26 tl2 kernel: RIP [<ffffffffa05d0ea4>] put_layout >>> +0x2f/0xa7 [nfs] >>> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20> >>> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]--- >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux- >>> nfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" >> in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 2010-06-28 18:53 ` Benny Halevy @ 2010-06-28 19:22 ` William A. (Andy) Adamson [not found] ` <AANLkTilDLWK8rfwzlI8xJJUckxljCqgmblAYj9ANOMnb-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: William A. (Andy) Adamson @ 2010-06-28 19:22 UTC (permalink / raw) To: Benny Halevy; +Cc: linux-nfs@vger.kernel.org Mailing list On Mon, Jun 28, 2010 at 2:53 PM, Benny Halevy <bhalevy@panasas.com> wro= te: > On Jun. 28, 2010, 19:44 +0300, Andy Adamson <andros@netapp.com> wrote= : >> Hi Benny >> >> I have not been able to reproduce this BUG. I've tried against the >> files pyNFS server with return_on_close False as well a True, and >> against a GFS2/pNFS cluster with write layouts turned on. >> >> Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch >> calls put_lseg when I/O to a DS fails. I tested this using the pyNFS >> files layout server and blocking the DS with iptables. I think this = is >> the only change in this patch set that would affect the refcounting. >> >> Are you able to reproduce the BUG? > > The easiest way I found to reproduce this bug is running the cthon te= sts > on a locally mounted file system exported over PNFSD_LOCAL_EXPORT. > The test machine is a dual core SMP machine. > Are you testing over a VM? =A0Is it uni-processor? Its a VM with one processor, but with SMP support turned on in the kernel. I just added a processor and will try re-running tests. -->Andy > > Benny > >> >> -->Andy >> >> On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote: >> >>> OK - I'll look into it. >>> >>> Sorry I missed today's pNFS call. >>> >>> -->Andy >>> >>> On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com> >>> wrote: >>>> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote: >>>>> Responded to comments, added a 2 cleanup patchses >>>>> >>>>> Plus some code cleanup >>>>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch >>>>> >>>>> and some bug fixes >>>>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch >>>>> >>>>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-th= e- >>>>> MDS.patch >>>>> was replaced by: >>>>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch >>>>> >>>>> >>>>> Remove unused (by file layout) encode_layoutreturn io operation >>>>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch >>>>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch >>>>> >>>>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch >>>>> >>>>> Note: pnfs4_proc_layoutget is only called by send_layout() which >>>>> prints >>>>> the status. >>>>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch >>>>> >>>>> Add back encode_layoutreturn io operation >>>>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch >>>>> >>>>> >>>>> New patches: >>>>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch >>>>> >>>>> This gets rid of a frame stack warning; >>>>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch >>>>> >>>>> Testing: >>>>> --------- >>>>> >>>>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS >>>>> Passes Connectathon tests >>>>> >>>>> Tested layoutget and layoutreturn recovery from >>>>> NFS4ERR_DEAD_SESSION with the >>>>> pyNFS server and the testclient framework. >>>>> >>>>> Still todo: >>>>> >>>>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, >>>>> layoutget, and >>>>> layoutcommit do not pass nfs_stste to the error handlers. >>>>> >>>>> Handle NFS4ERR_BAD_LAYOUT. >>>>> >>>>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. >>>>> >>>>> -->Andy >>>> >>>> Andy, I've hit >>>> =A0 =A0 =A0 BUG_ON(lo->refcount <=3D 0); >>>> in put_layout() with this patchset. >>>> I'm not sure if it introduced it or not, still investigating... >>>> >>>> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: >>>> layout.refcount 1 >>>> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------ >>>> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalevy= / >>>> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341! >>>> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP >>>> DEBUG_PAGEALLOC >>>> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/ >>>> initstate >>>> Jun 24 12:07:26 tl2 kernel: CPU 1 >>>> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nfs= d >>>> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 >>>> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi >>>> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror >>>> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm >>>> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel >>>> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core >>>> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore >>>> snd_page_alloc button video output rng_core sg cdrom ata_generic >>>> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd >>>> ohci_hcd ehci_hcd [last unloaded: microcode] >>>> Jun 24 12:07:26 tl2 kernel: >>>> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted >>>> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527 >>>> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>] >>>> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] >>>> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 =A0EFLAGS: >>>> 00010246 >>>> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: >>>> ffff8800704b6b78 RCX: 0000000000000066 >>>> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: >>>> ffffea0001b931a8 RDI: ffff8800704b6b78 >>>> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: >>>> 0000000000000000 R09: ffff88007356a500 >>>> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: >>>> 0000000000000003 R12: ffff8800704b69a8 >>>> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: >>>> ffff88007356a508 R15: ffff88007356a590 >>>> Jun 24 12:07:26 tl2 kernel: FS: =A00000000000000000(0000) >>>> GS:ffff880001a80000(0000) knlGS:0000000000000000 >>>> Jun 24 12:07:26 tl2 kernel: CS: =A00010 DS: 0000 ES: 0000 CR0: >>>> 000000008005003b >>>> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: >>>> 0000000001698000 CR4: 00000000000406e0 >>>> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: >>>> 0000000000000000 DR2: 0000000000000000 >>>> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: >>>> 00000000ffff0ff0 DR7: 0000000000000400 >>>> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadinf= o >>>> ffff88007525c000, task ffff88007d988000) >>>> Jun 24 12:07:26 tl2 kernel: Stack: >>>> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 >>>> ffff88007525dd60 ffffffffa05d203f >>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 >>>> ffff880073854f00 ffffffffa05d5880 >>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c >>>> ffff88007525dd90 ffff88007356a500 >>>> Jun 24 12:07:26 tl2 kernel: Call Trace: >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_relea= se >>>> +0x43/0x68 [nfs] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] >>>> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] >>>> rpc_release_calldata+0x17/0x19 [sunrpc] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x5= e/ >>>> 0x66 [sunrpc] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task >>>> +0x98/0x9c [sunrpc] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute >>>> +0x205/0x212 [sunrpc] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedul= e >>>> +0x15/0x17 [sunrpc] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread >>>> +0x1aa/0x23b >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? >>>> rpc_async_schedule+0x0/0x17 [sunrpc] >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? >>>> autoremove_wake_function+0x0/0x39 >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? >>>> spin_unlock_irqrestore+0xe/0x10 >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread >>>> +0x0/0x23b >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x87 >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] >>>> kernel_thread_helper+0x4/0x10 >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x8= 7 >>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? >>>> kernel_thread_helper+0x0/0x10 >>>> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 24 >>>> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 >>>> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 6= 7 >>>> 48 8b 82 48 03 00 00 f6 05 >>>> Jun 24 12:07:26 tl2 kernel: RIP =A0[<ffffffffa05d0ea4>] put_layout >>>> +0x2f/0xa7 [nfs] >>>> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20> >>>> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]--- >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux- >>>> nfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.ht= ml >>>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-nfs= " >>> in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm= l >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs"= in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" = in > the body of a message to majordomo@vger.kernel.org > More majordomo info at =A0http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <AANLkTilDLWK8rfwzlI8xJJUckxljCqgmblAYj9ANOMnb-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 [not found] ` <AANLkTilDLWK8rfwzlI8xJJUckxljCqgmblAYj9ANOMnb-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2010-06-28 20:02 ` William A. (Andy) Adamson 0 siblings, 0 replies; 20+ messages in thread From: William A. (Andy) Adamson @ 2010-06-28 20:02 UTC (permalink / raw) To: Benny Halevy; +Cc: linux-nfs@vger.kernel.org Mailing list On Mon, Jun 28, 2010 at 3:22 PM, William A. (Andy) Adamson <androsadamson@gmail.com> wrote: > On Mon, Jun 28, 2010 at 2:53 PM, Benny Halevy <bhalevy@panasas.com> w= rote: >> On Jun. 28, 2010, 19:44 +0300, Andy Adamson <andros@netapp.com> wrot= e: >>> Hi Benny >>> >>> I have not been able to reproduce this BUG. I've tried against the >>> files pyNFS server with return_on_close False as well a True, and >>> against a GFS2/pNFS cluster with write layouts turned on. >>> >>> Patch 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patc= h >>> calls put_lseg when I/O to a DS fails. I tested this using the pyNF= S >>> files layout server and blocking the DS with iptables. I think this= is >>> the only change in this patch set that would affect the refcounting= =2E >>> >>> Are you able to reproduce the BUG? >> >> The easiest way I found to reproduce this bug is running the cthon t= ests >> on a locally mounted file system exported over PNFSD_LOCAL_EXPORT. >> The test machine is a dual core SMP machine. >> Are you testing over a VM? =A0Is it uni-processor? > > Its a VM with one processor, but with SMP support turned on in the > kernel. I just added a processor and will try re-running tests. Added a processor - all cthon tests succeeded. Just to be clear, I'm testing the client pnfs-submit branch. -->Andy > > -->Andy > >> >> Benny >> >>> >>> -->Andy >>> >>> On Jun 24, 2010, at 1:02 PM, William A. (Andy) Adamson wrote: >>> >>>> OK - I'll look into it. >>>> >>>> Sorry I missed today's pNFS call. >>>> >>>> -->Andy >>>> >>>> On Thu, Jun 24, 2010 at 9:14 AM, Benny Halevy <bhalevy@panasas.com= > >>>> wrote: >>>>> On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote: >>>>>> Responded to comments, added a 2 cleanup patchses >>>>>> >>>>>> Plus some code cleanup >>>>>> 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch >>>>>> >>>>>> and some bug fixes >>>>>> 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch >>>>>> >>>>>> NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-t= he- >>>>>> MDS.patch >>>>>> was replaced by: >>>>>> 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch >>>>>> >>>>>> >>>>>> Remove unused (by file layout) encode_layoutreturn io operation >>>>>> 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch >>>>>> 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch >>>>>> >>>>>> 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch >>>>>> >>>>>> Note: pnfs4_proc_layoutget is only called by send_layout() which >>>>>> prints >>>>>> the status. >>>>>> 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch >>>>>> >>>>>> Add back encode_layoutreturn io operation >>>>>> 0008-pnfs-post-submit-restore-encode_layoutreturn.patch >>>>>> >>>>>> >>>>>> New patches: >>>>>> 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch >>>>>> >>>>>> This gets rid of a frame stack warning; >>>>>> 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch >>>>>> >>>>>> Testing: >>>>>> --------- >>>>>> >>>>>> CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS >>>>>> Passes Connectathon tests >>>>>> >>>>>> Tested layoutget and layoutreturn recovery from >>>>>> NFS4ERR_DEAD_SESSION with the >>>>>> pyNFS server and the testclient framework. >>>>>> >>>>>> Still todo: >>>>>> >>>>>> Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, >>>>>> layoutget, and >>>>>> layoutcommit do not pass nfs_stste to the error handlers. >>>>>> >>>>>> Handle NFS4ERR_BAD_LAYOUT. >>>>>> >>>>>> CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. >>>>>> >>>>>> -->Andy >>>>> >>>>> Andy, I've hit >>>>> =A0 =A0 =A0 BUG_ON(lo->refcount <=3D 0); >>>>> in put_layout() with this patchset. >>>>> I'm not sure if it introduced it or not, still investigating... >>>>> >>>>> Jun 24 12:07:26 tl2 kernel: pnfs_destroy_inode: WARNING: >>>>> layout.refcount 1 >>>>> Jun 24 12:07:26 tl2 kernel: ------------[ cut here ]------------ >>>>> Jun 24 12:07:26 tl2 kernel: kernel BUG at /usr0/export/dev/bhalev= y/ >>>>> git/linux-pnfs-bh-nfs41/fs/nfs/pnfs.c:341! >>>>> Jun 24 12:07:26 tl2 kernel: invalid opcode: 0000 [#1] SMP >>>>> DEBUG_PAGEALLOC >>>>> Jun 24 12:07:26 tl2 kernel: last sysfs file: /sys/module/nfs/ >>>>> initstate >>>>> Jun 24 12:07:26 tl2 kernel: CPU 1 >>>>> Jun 24 12:07:26 tl2 kernel: Modules linked in: nfslayoutdriver nf= sd >>>>> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc osd libosd autofs4 >>>>> crc32c ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi >>>>> cpufreq_ondemand acpi_cpufreq freq_table mperf ext3 jbd dm_mirror >>>>> dm_region_hash dm_log dm_multipath dm_mod kvm_intel kvm >>>>> snd_hda_codec_realtek i915 drm_kms_helper drm snd_hda_intel >>>>> snd_hda_codec snd_hwdep i2c_algo_bit snd_seq i2c_i801 i2c_core >>>>> snd_seq_device snd_pcm r8169 mii snd_timer sr_mod snd soundcore >>>>> snd_page_alloc button video output rng_core sg cdrom ata_generic >>>>> ata_piix libata sd_mod scsi_mod ext4 mbcache jbd2 crc16 uhci_hcd >>>>> ohci_hcd ehci_hcd [last unloaded: microcode] >>>>> Jun 24 12:07:26 tl2 kernel: >>>>> Jun 24 12:07:26 tl2 kernel: Pid: 1920, comm: rpciod/1 Not tainted >>>>> 2.6.35-rc3-pnfs+ #54 G31M4 (MS-7527)/MS-7527 >>>>> Jun 24 12:07:26 tl2 kernel: RIP: 0010:[<ffffffffa05d0ea4>] >>>>> [<ffffffffa05d0ea4>] put_layout+0x2f/0xa7 [nfs] >>>>> Jun 24 12:07:26 tl2 kernel: RSP: 0018:ffff88007525dd20 =A0EFLAGS: >>>>> 00010246 >>>>> Jun 24 12:07:26 tl2 kernel: RAX: 0000000000000000 RBX: >>>>> ffff8800704b6b78 RCX: 0000000000000066 >>>>> Jun 24 12:07:26 tl2 kernel: RDX: ffff8800704b69a8 RSI: >>>>> ffffea0001b931a8 RDI: ffff8800704b6b78 >>>>> Jun 24 12:07:26 tl2 kernel: RBP: ffff88007525dd30 R08: >>>>> 0000000000000000 R09: ffff88007356a500 >>>>> Jun 24 12:07:26 tl2 kernel: R10: ffff88007525dd80 R11: >>>>> 0000000000000003 R12: ffff8800704b69a8 >>>>> Jun 24 12:07:26 tl2 kernel: R13: ffff880073854f00 R14: >>>>> ffff88007356a508 R15: ffff88007356a590 >>>>> Jun 24 12:07:26 tl2 kernel: FS: =A00000000000000000(0000) >>>>> GS:ffff880001a80000(0000) knlGS:0000000000000000 >>>>> Jun 24 12:07:26 tl2 kernel: CS: =A00010 DS: 0000 ES: 0000 CR0: >>>>> 000000008005003b >>>>> Jun 24 12:07:26 tl2 kernel: CR2: 0000003944279000 CR3: >>>>> 0000000001698000 CR4: 00000000000406e0 >>>>> Jun 24 12:07:26 tl2 kernel: DR0: 0000000000000000 DR1: >>>>> 0000000000000000 DR2: 0000000000000000 >>>>> Jun 24 12:07:26 tl2 kernel: DR3: 0000000000000000 DR6: >>>>> 00000000ffff0ff0 DR7: 0000000000000400 >>>>> Jun 24 12:07:26 tl2 kernel: Process rpciod/1 (pid: 1920, threadin= fo >>>>> ffff88007525c000, task ffff88007d988000) >>>>> Jun 24 12:07:26 tl2 kernel: Stack: >>>>> Jun 24 12:07:26 tl2 kernel: ffff8800704b6b78 ffff8800704b69a8 >>>>> ffff88007525dd60 ffffffffa05d203f >>>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd60 ffff880073854f18 >>>>> ffff880073854f00 ffffffffa05d5880 >>>>> Jun 24 12:07:26 tl2 kernel: <0> ffff88007525dd80 ffffffffa05bfb5c >>>>> ffff88007525dd90 ffff88007356a500 >>>>> Jun 24 12:07:26 tl2 kernel: Call Trace: >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05d203f>] pnfs_layout_rele= ase >>>>> +0x43/0x68 [nfs] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05bfb5c>] >>>>> nfs4_pnfs_layoutreturn_release+0x61/0x8b [nfs] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056207d>] >>>>> rpc_release_calldata+0x17/0x19 [sunrpc] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa05621bd>] rpc_free_task+0x= 5e/ >>>>> 0x66 [sunrpc] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa056225d>] rpc_put_task >>>>> +0x98/0x9c [sunrpc] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ea7>] __rpc_execute >>>>> +0x205/0x212 [sunrpc] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562ef0>] rpc_async_schedu= le >>>>> +0x15/0x17 [sunrpc] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052cb7>] worker_thread >>>>> +0x1aa/0x23b >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffffa0562edb>] ? >>>>> rpc_async_schedule+0x0/0x17 [sunrpc] >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056ab7>] ? >>>>> autoremove_wake_function+0x0/0x39 >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff8102f96d>] ? >>>>> spin_unlock_irqrestore+0xe/0x10 >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81052b0d>] ? worker_thread >>>>> +0x0/0x23b >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81056645>] kthread+0x7f/0x8= 7 >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a24>] >>>>> kernel_thread_helper+0x4/0x10 >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff810565c6>] ? kthread+0x0/0x= 87 >>>>> Jun 24 12:07:26 tl2 kernel: [<ffffffff81003a20>] ? >>>>> kernel_thread_helper+0x0/0x10 >>>>> Jun 24 12:07:26 tl2 kernel: Code: 41 54 53 0f 1f 44 00 00 8b 87 2= 4 >>>>> 01 00 00 48 89 fb 48 8d 97 30 fe ff ff 89 c1 c1 f9 08 38 c1 75 04 >>>>> 0f 0b eb fe 8b 07 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 07 75 = 67 >>>>> 48 8b 82 48 03 00 00 f6 05 >>>>> Jun 24 12:07:26 tl2 kernel: RIP =A0[<ffffffffa05d0ea4>] put_layou= t >>>>> +0x2f/0xa7 [nfs] >>>>> Jun 24 12:07:27 tl2 kernel: RSP <ffff88007525dd20> >>>>> Jun 24 12:07:27 tl2 kernel: ---[ end trace 0468384c0ab45a1f ]--- >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux- >>>>> nfs" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.h= tml >>>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-nf= s" >>>> in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.ht= ml >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-nfs= " in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm= l >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs"= in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html >> > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 2010-06-23 19:21 [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 andros 2010-06-23 19:21 ` [PATCH 01/10] SQUASHME: pnfs-submit remove unused filelayout_mount_type andros 2010-06-24 13:14 ` [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 Benny Halevy @ 2010-07-01 18:27 ` Benny Halevy 2 siblings, 0 replies; 20+ messages in thread From: Benny Halevy @ 2010-07-01 18:27 UTC (permalink / raw) To: andros; +Cc: linux-nfs On 2010-06-23 22:21, andros@netapp.com wrote:On Jun. 23, 2010, 22:21 +0300, andros@netapp.com wrote: > Responded to comments, added a 2 cleanup patchses > > Plus some code cleanup > 0001-SQUASHME-pnfs-submit-remove-unused-filelayout_mount_.patch > > and some bug fixes > 0002-SQUASHME-pnfs-submit-pnfs_try_to_read-write-commit-u.patch > > NOTE: this patch: 0003-SQUASHME-pnfs-submit-tell-commit-to-use-the-MDS.patch > was replaced by: > 0003-SQUASHME-pnfs-submit-clear-page-lseg-on-partial-i-o.patch > All but patch 3/10 (as per your request) are merged at pnfs-all-2.6.35-rc3-2010-07-01 Thanks! Benny > > Remove unused (by file layout) encode_layoutreturn io operation > 0004-SQUASHME-pnfs-submit-remove-encode_layoutreturn.patch > 0005-SQUASHME-pnfs-submit-add-error-handling-to-layout-re.patch > > 0006-SQUASHME-pnfs-submit-handle-assassinated-layoutcommi.patch > > Note: pnfs4_proc_layoutget is only called by send_layout() which prints > the status. > 0007-SQUASHME-pnfs-submit-add-error-handlers-to-layout-ge.patch > > Add back encode_layoutreturn io operation > 0008-pnfs-post-submit-restore-encode_layoutreturn.patch > > > New patches: > 0009-SQUASHME-pnfs-submit-don-t-re-initialize-i_lock.patch > > This gets rid of a frame stack warning; > 0010-SQUASHME-pnfs-submit-remove-struct-nfs_server-from-s.patch > > Testing: > --------- > > CONFIG_NFS_V4_1 set: NFSv4.0 NFSv4.1 pNFS > Passes Connectathon tests > > Tested layoutget and layoutreturn recovery from NFS4ERR_DEAD_SESSION with the > pyNFS server and the testclient framework. > > Still todo: > > Recover from NFS4ERR_BAD_STATEID. Currently layoutreturn, layoutget, and > layoutcommit do not pass nfs_stste to the error handlers. > > Handle NFS4ERR_BAD_LAYOUT. > > CONFIG_NFS_V4_1 not set: NFSv4.o mount passes cthon tests. > > -->Andy -- Benny Halevy Software Architect Panasas, Inc. bhalevy@panasas.com Tel/Fax: +972-3-647-8340 Mobile: +972-54-802-8340 Panasas: The Leader in Parallel Storage www.panasas.com ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2010-07-01 18:27 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-06-23 19:21 [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 andros 2010-06-23 19:21 ` [PATCH 01/10] SQUASHME: pnfs-submit remove unused filelayout_mount_type andros 2010-06-23 19:21 ` [PATCH 02/10] SQUASHME pnfs-submit: pnfs_try_to_read, write, commit using freed memory andros 2010-06-23 19:21 ` [PATCH 03/10] SQUASHME pnfs-submit: clear page lseg on partial i/o andros 2010-06-23 19:21 ` [PATCH 04/10] SQUASHME pnfs-submit: remove encode_layoutreturn andros 2010-06-23 19:21 ` [PATCH 05/10] SQUASHME pnfs-submit: add error handling to layout return andros 2010-06-23 19:21 ` [PATCH 06/10] SQUASHME pnfs-submit: handle assassinated layoutcommit andros 2010-06-23 19:21 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get andros 2010-06-23 19:21 ` [PATCH 08/10] pnfs-post-submit: restore encode_layoutreturn andros 2010-06-23 19:21 ` [PATCH 09/10] SQUASHME: pnfs-submit: don't re-initialize i_lock andros 2010-06-23 19:21 ` [PATCH 10/10] SQUASHME pnfs-submit: remove struct nfs_server from stack andros 2010-06-30 15:19 ` [PATCH 07/10] SQUASHME pnfs-submit: add error handlers to layout get Boaz Harrosh 2010-06-30 19:23 ` William A. (Andy) Adamson 2010-06-24 13:14 ` [PATCH 0/10] pnfs-submit add layoutget,layoutreturn error handling version 2 Benny Halevy 2010-06-24 17:02 ` William A. (Andy) Adamson [not found] ` <AANLkTikJWftkWhU8TIOGxvGxo8s2_sXyMn8VIsk9caTv-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2010-06-28 16:44 ` Andy Adamson 2010-06-28 18:53 ` Benny Halevy 2010-06-28 19:22 ` William A. (Andy) Adamson [not found] ` <AANLkTilDLWK8rfwzlI8xJJUckxljCqgmblAYj9ANOMnb-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2010-06-28 20:02 ` William A. (Andy) Adamson 2010-07-01 18:27 ` Benny Halevy
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.