[PATCH 0/14] pnfs_submit file layout client device management

[PATCH 0/14] pnfs_submit file layout client device management

[andros]

Apply against 2.6.34-rc3 pnfs branch. For pNFS client submission. Only touches
the file layout driver.

This set of patches implements several design changes enmerated at
http://wiki.linux-nfs.org/wiki/index.php/Porposed_Device_ Manangement_Design.

- Only call GETDEVICEINFO from one place
- Only cache the first data server in the multipath_list4 array.
- Wait to connect to data server when needed for I/O
- Move data server cache to stand alone cache

Also some cleanup, and function interfaces are changed in 
preparation for the soon to come generic deviceid cache.

0001-SQUASHME-pnfs_submit-remove-unused-getdevicelist-fun.patch
0002-SQUASHME-pnfs-submit-call-get_device_info-from-filel.patch
0003-SQUASHME-pnfs_submit-remove-multilist4-caching.patch
0004-SQUASHME-pnfs_submit-wait-for-I-O-for-data-server-co.patch
0005-SQUASHME-pnfs_submit-remove-xdr-macros.patch
0006-SQUASHME-pnfs_submit-remove-unused-nfs4_pnfs_device_.patch
0007-SQUASHME-pnfs_submit-change-get_device_info-paramete.patch
0008-SQUASHME-pnfs_submit-change-decode_and_add_device-pa.patch
0009-SQUASHME-pnfs_submit-change-decode_device-parameter.patch
0010-SQUASHME-pnfs_submit-change-decode_and_add_ds-parame.patch
0011-SQUASHME-pnfs_submit-change-nfs4_pnfs_ds_add-paramet.patch
0012-SQUASHME-pnfs_submit-remove-filelayout-CB_NOTIFY_DEV.patch
0013-SQUASHME-pnfs_submit-stand-alone-data-server-cache.patch
0014-SQUASHME-pnfs_submit-remove-kfree-from-under-spinloc.patch

Testing:
CONFIG_NFS_V4_1 set:
pNFS mount, NFSv4.1 mount, NFSv4.0 mount:
Connecatathon tests pass. pyNFS testclient tests pass.

CONFIG_NFS_V4_1 not set:
Connecatahon tests pass.


-->Andy

[PATCH 01/14] SQUASHME: pnfs_submit: remove unused getdevicelist functions

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.h    |    4 ----
 fs/nfs/nfs4filelayoutdev.c |   27 ---------------------------
 2 files changed, 0 insertions(+), 31 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index 6051195..d003809 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -106,10 +106,6 @@ int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 			  loff_t offset,
 			  size_t count,
 			  struct nfs4_pnfs_dserver *dserver);
-int decode_and_add_devicelist(struct filelayout_mount_type *mt, struct pnfs_devicelist *devlist);
-int process_deviceid_list(struct filelayout_mount_type *mt,
-			  struct nfs_fh *fh,
-			  struct pnfs_devicelist *devlist);
 u32 filelayout_dserver_get_index(loff_t offset,
 				 struct nfs4_file_layout_dsaddr *di,
 				 struct nfs4_filelayout_segment *layout);
diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 16dafc0..e557679 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -662,33 +662,6 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
 	return dsaddr;
 }
 
-/* For each deviceid, if not already in the cache,
- * call getdeviceinfo and add the devices associated with
- * the deviceid to the list of available devices for this
- * mount point.
- * Must at some point be followed up with nfs4_pnfs_device_destroy.
- */
-int
-process_deviceid_list(struct filelayout_mount_type *mt,
-		      struct nfs_fh *fh,
-		      struct pnfs_devicelist *devlist)
-{
-	int i;
-
-	dprintk("--> %s: num_devs=%d\n", __func__, devlist->num_devs);
-
-	for (i = 0; i < devlist->num_devs; i++) {
-		if (!nfs4_file_layout_dsaddr_get(mt, &devlist->dev_id[i])) {
-			printk(KERN_WARNING
-			       "<-- %s: Error retrieving device %d\n",
-			       __func__, i);
-			return 1;
-		}
-	}
-	dprintk("<-- %s: success\n", __func__);
-	return 0;
-}
-
 /* Retrieve the information for dev_id, add it to the list
  * of available devices, and return it.
  */
-- 
1.6.6

[PATCH 02/14] SQUASHME pnfs-submit: call get_device_info from filelayout_check

[andros]

From: Andy Adamson <andros@netapp.com>

filelayout_check() validates a LAYOUTGET return including the deviceid.
If the deviceid is not cached, GETDEVICEINFO is called.

Once cached, the deviceid will only be uncached upon umount or a return of all
refering layouts. So, there is no need for a get_device_info call in the I/O
paths.

Remove nfs4_file_layout_dsaddr_get.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.c    |   11 +++++++----
 fs/nfs/nfs4filelayout.h    |    6 ++++--
 fs/nfs/nfs4filelayoutdev.c |   27 +++------------------------
 3 files changed, 14 insertions(+), 30 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
index df6c1d0..ffc3ef2 100644
--- a/fs/nfs/nfs4filelayout.c
+++ b/fs/nfs/nfs4filelayout.c
@@ -413,12 +413,15 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	struct nfs_server *nfss = NFS_SERVER(PNFS_INODE(lo));
 
 	dprintk("--> %s\n", __func__);
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(PNFS_INODE(lo)),
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
 					     &fl->dev_id);
 	if (dsaddr == NULL) {
-		dprintk("%s NO device for dev_id %s\n",
+		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
+		if (dsaddr == NULL) {
+			dprintk("%s NO device for dev_id %s\n",
 				__func__, deviceid_fmt(&fl->dev_id));
-		goto out;
+			goto out;
+		}
 	}
 	if (fl->first_stripe_index < 0 ||
 	    fl->first_stripe_index > dsaddr->stripe_count) {
@@ -645,7 +648,7 @@ filelayout_commit(struct pnfs_layout_type *layoutid, int sync,
 	stripesz = filelayout_get_stripesize(layoutid);
 	dprintk("%s stripesize %Zd\n", __func__, stripesz);
 
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(data->inode),
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(data->inode)->hlist,
 					     &nfslay->dev_id);
 	if (dsaddr == NULL) {
 		data->pdata.pnfs_error = -EIO;
diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index d003809..3821989 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -121,7 +121,9 @@ u32 filelayout_dserver_get_index(loff_t offset,
 } while (0)
 
 struct nfs4_file_layout_dsaddr *
-nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *,
-			  struct pnfs_deviceid *);
+nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
+			   struct pnfs_deviceid *dev_id);
+struct nfs4_file_layout_dsaddr *
+get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
 
 #endif /* FS_NFS_NFS4FILELAYOUT_H */
diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index e557679..ebb9845 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -52,13 +52,6 @@
 
 #define NFSDBG_FACILITY		NFSDBG_PNFS_LD
 
-struct nfs4_file_layout_dsaddr *nfs4_file_layout_dsaddr_get(
-					struct filelayout_mount_type *mt,
-					struct pnfs_deviceid *dev_id);
-struct nfs4_file_layout_dsaddr *nfs4_pnfs_device_item_find(
-					struct nfs4_pnfs_dev_hlist *hlist,
-					struct pnfs_deviceid *dev_id);
-
 void
 print_ds_list(struct nfs4_multipath *multipath)
 {
@@ -665,7 +658,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
 /* Retrieve the information for dev_id, add it to the list
  * of available devices, and return it.
  */
-static struct nfs4_file_layout_dsaddr *
+struct nfs4_file_layout_dsaddr *
 get_device_info(struct filelayout_mount_type *mt,
 		struct pnfs_deviceid *dev_id)
 {
@@ -737,21 +730,6 @@ out_free:
 }
 
 struct nfs4_file_layout_dsaddr *
-nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *mt,
-			    struct pnfs_deviceid *dev_id)
-{
-	struct nfs4_file_layout_dsaddr *dsaddr;
-
-	read_lock(&mt->hlist->dev_lock);
-	dsaddr = _device_lookup(mt->hlist, dev_id);
-	read_unlock(&mt->hlist->dev_lock);
-
-	if (dsaddr == NULL)
-		dsaddr = get_device_info(mt, dev_id);
-	return dsaddr;
-}
-
-struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id)
 {
@@ -798,7 +776,8 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	if (!layout)
 		return 1;
 
-	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(inode), &layout->dev_id);
+	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(inode)->hlist,
+					    &layout->dev_id);
 	if (dsaddr == NULL)
 		return 1;
 
-- 
1.6.6

[PATCH 03/14] SQUASHME pnfs_submit: remove multilist4 caching

[andros]

From: Andy Adamson <andros@netapp.com>

We only support the use of the first multipath4 data server.

Remove the multipath structures from struct nfs4_file_layout_dsaddr
and regroup it's allocation to fit the new strucuture.

Refactor the associated debug print routines.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.c    |   44 ++++---------
 fs/nfs/nfs4filelayout.h    |   12 +---
 fs/nfs/nfs4filelayoutdev.c |  151 +++++++++++++++++++-------------------------
 3 files changed, 81 insertions(+), 126 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
index ffc3ef2..abdf691 100644
--- a/fs/nfs/nfs4filelayout.c
+++ b/fs/nfs/nfs4filelayout.c
@@ -252,7 +252,6 @@ filelayout_read_pagelist(struct pnfs_layout_type *layoutid,
 	struct inode *inode = PNFS_INODE(layoutid);
 	struct nfs4_filelayout_segment *flseg;
 	struct nfs4_pnfs_dserver dserver;
-	struct nfs4_pnfs_ds *ds;
 	int status;
 
 	dprintk("--> %s ino %lu nr_pages %d pgbase %u req %Zu@%llu\n",
@@ -271,14 +270,12 @@ filelayout_read_pagelist(struct pnfs_layout_type *layoutid,
 		return PNFS_NOT_ATTEMPTED;
 	}
 
-	ds = dserver.multipath->ds_list[0];
-
 	dprintk("%s USE DS:ip %x %s\n", __func__,
-		htonl(ds->ds_ip_addr), ds->r_addr);
+		htonl(dserver.ds->ds_ip_addr), dserver.ds->r_addr);
 
 	/* just try the first data server for the index..*/
-	data->fldata.pnfs_client = ds->ds_clp->cl_rpcclient;
-	data->fldata.ds_nfs_client = ds->ds_clp;
+	data->fldata.pnfs_client = dserver.ds->ds_clp->cl_rpcclient;
+	data->fldata.ds_nfs_client = dserver.ds->ds_clp;
 	data->args.fh = dserver.fh;
 
 	/* Now get the file offset on the dserver
@@ -300,16 +297,6 @@ filelayout_read_pagelist(struct pnfs_layout_type *layoutid,
 	return PNFS_ATTEMPTED;
 }
 
-void
-print_ds(struct nfs4_pnfs_ds *ds)
-{
-	dprintk("        ds->ds_ip_addr %x\n", htonl(ds->ds_ip_addr));
-	dprintk("        ds->ds_port %hu\n", ntohs(ds->ds_port));
-	dprintk("        ds->ds_clp %p\n", ds->ds_clp);
-	dprintk("        ds->ds_count %d\n", atomic_read(&ds->ds_count));
-	dprintk("        %s\n", ds->r_addr);
-}
-
 /* Perform async writes. */
 static enum pnfs_try_status
 filelayout_write_pagelist(struct pnfs_layout_type *layoutid,
@@ -324,7 +311,6 @@ filelayout_write_pagelist(struct pnfs_layout_type *layoutid,
 	struct inode *inode = PNFS_INODE(layoutid);
 	struct nfs4_filelayout_segment *flseg = LSEG_LD_DATA(data->pdata.lseg);
 	struct nfs4_pnfs_dserver dserver;
-	struct nfs4_pnfs_ds *ds;
 	int status;
 
 	dprintk("--> %s ino %lu nr_pages %d pgbase %u req %Zu@%llu sync %d\n",
@@ -342,16 +328,13 @@ filelayout_write_pagelist(struct pnfs_layout_type *layoutid,
 		return PNFS_NOT_ATTEMPTED;
 	}
 
-	/* use the first multipath data server */
-	ds = dserver.multipath->ds_list[0];
-
 	dprintk("%s ino %lu %Zu@%llu DS:%x:%hu %s\n",
 		__func__, inode->i_ino, count, offset,
-		htonl(ds->ds_ip_addr), ntohs(ds->ds_port),
-		ds->r_addr);
+		htonl(dserver.ds->ds_ip_addr), ntohs(dserver.ds->ds_port),
+		dserver.ds->r_addr);
 
-	data->fldata.pnfs_client = ds->ds_clp->cl_rpcclient;
-	data->fldata.ds_nfs_client = ds->ds_clp;
+	data->fldata.pnfs_client = dserver.ds->ds_clp->cl_rpcclient;
+	data->fldata.ds_nfs_client = dserver.ds->ds_clp;
 	data->args.fh = dserver.fh;
 
 	/* Get the file offset on the dserver. Set the write offset to
@@ -396,8 +379,8 @@ filelayout_free_layout(void *layoutid)
  * Make sure layout segment parameters are sane WRT the device.
  *
  * Notes:
- * 1) current code insists that # stripe index = # multipath devices which
- *    is wrong.
+ * 1) current code insists that # stripe index = # data servers in ds_list
+ *    which is wrong.
  * 2) pattern_offset is ignored and must == 0 which is wrong;
  * 3) the pattern_offset needs to be a mutliple of the stripe unit.
  * 4) stripe unit is multiple of page size
@@ -443,9 +426,9 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	}
 
 	/* XXX only support SPARSE packing. Don't support use MDS open fh */
-	if (!(fl->num_fh == 1 || fl->num_fh == dsaddr->multipath_count)) {
-		dprintk("%s num_fh %u not equal to 1 or multipath_count %u\n",
-			__func__, fl->num_fh, dsaddr->multipath_count);
+	if (!(fl->num_fh == 1 || fl->num_fh == dsaddr->ds_num)) {
+		dprintk("%s num_fh %u not equal to 1 or ds_num %u\n",
+			__func__, fl->num_fh, dsaddr->ds_num);
 		goto out;
 	}
 
@@ -714,8 +697,7 @@ filelayout_commit(struct pnfs_layout_type *layoutid, int sync,
 		list_add(&dsdata->pages, &head2);
 		list_del_init(&head2);
 
-		/* Just try the first multipath data server */
-		ds = dserver.multipath->ds_list[0];
+		ds = dserver.ds;
 		dsdata->fldata.pnfs_client = ds->ds_clp->cl_rpcclient;
 		dsdata->fldata.ds_nfs_client = ds->ds_clp;
 		dsdata->args.fh = dserver.fh;
diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index 3821989..0df1eaf 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -42,18 +42,13 @@ struct nfs4_pnfs_ds {
 	char r_addr[29];
 };
 
-struct nfs4_multipath {
-	int 			num_ds;
-	struct nfs4_pnfs_ds	*ds_list[NFS4_PNFS_MAX_MULTI_DS];
-};
-
 struct nfs4_file_layout_dsaddr {
 	struct hlist_node	hash_node;   /* nfs4_pnfs_dev_hlist dev_list */
 	struct pnfs_deviceid	dev_id;
 	u32 			stripe_count;
 	u8			*stripe_indices;
-	u32			multipath_count;
-	struct nfs4_multipath	*multipath_list;
+	u32			ds_num;
+	struct nfs4_pnfs_ds	*ds_list[1];
 };
 
 struct nfs4_pnfs_dev_hlist {
@@ -69,7 +64,7 @@ struct nfs4_pnfs_dev_hlist {
 
 struct nfs4_pnfs_dserver {
 	struct nfs_fh        *fh;
-	struct nfs4_multipath *multipath;
+	struct nfs4_pnfs_ds	*ds;
 };
 
 struct nfs4_filelayout_segment {
@@ -97,6 +92,7 @@ struct filelayout_mount_type {
 
 extern struct pnfs_client_operations *pnfs_callback_ops;
 
+extern void print_ds(struct nfs4_pnfs_ds *ds);
 char *deviceid_fmt(const struct pnfs_deviceid *dev_id);
 int  nfs4_pnfs_devlist_init(struct nfs4_pnfs_dev_hlist *hlist);
 void nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist);
diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index ebb9845..ea25bd2 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -53,43 +53,31 @@
 #define NFSDBG_FACILITY		NFSDBG_PNFS_LD
 
 void
-print_ds_list(struct nfs4_multipath *multipath)
+print_ds(struct nfs4_pnfs_ds *ds)
 {
-	struct nfs4_pnfs_ds *ds;
-	int i;
-
-	ds = multipath->ds_list[0];
 	if (ds == NULL) {
 		dprintk("%s NULL device \n", __func__);
 		return;
 	}
-	for (i = 0; i < multipath->num_ds; i++) {
-		dprintk("        ip_addr %x\n", ntohl(ds->ds_ip_addr));
-		dprintk("        port %hu\n", ntohs(ds->ds_port));
-		dprintk("        client %p\n", ds->ds_clp);
-		dprintk("        ref count %d\n", atomic_read(&ds->ds_count));
-		if (ds->ds_clp)
-			dprintk("        cl_exchange_flags %x\n",
+	dprintk("        ip_addr %x\n", ntohl(ds->ds_ip_addr));
+	dprintk("        port %hu\n", ntohs(ds->ds_port));
+	dprintk("        client %p\n", ds->ds_clp);
+	dprintk("        ref count %d\n", atomic_read(&ds->ds_count));
+	if (ds->ds_clp)
+		dprintk("        cl_exchange_flags %x\n",
 					    ds->ds_clp->cl_exchange_flags);
-		dprintk("        ip:port %s\n", ds->r_addr);
-		ds++;
-	}
+	dprintk("        ip:port %s\n", ds->r_addr);
 }
 
 void
-print_multipath_list(struct nfs4_file_layout_dsaddr *dsaddr)
+print_ds_list(struct nfs4_file_layout_dsaddr *dsaddr)
 {
-	struct nfs4_multipath *multipath;
 	int i;
 
-	multipath = &dsaddr->multipath_list[0];
-	dprintk("%s dsaddr->multipath_count %d\n", __func__,
-		dsaddr->multipath_count);
-	for (i = 0; i < dsaddr->multipath_count; i++) {
-		dprintk("        num_ds %d\n", multipath->num_ds);
-		print_ds_list(multipath);
-		multipath++;
-	}
+	dprintk("%s dsaddr->ds_num %d\n", __func__,
+		dsaddr->ds_num);
+	for (i = 0; i < dsaddr->ds_num; i++)
+		print_ds(dsaddr->ds_list[i]);
 }
 
 /* Debugging function assuming a 64bit major/minor split of the deviceid */
@@ -174,9 +162,9 @@ _device_add(struct nfs4_pnfs_dev_hlist *hlist,
 {
 	unsigned long      hash;
 
-	dprintk("_device_add: dev_id=%s\nmultipath_list:\n",
+	dprintk("_device_add: dev_id=%s ds_list:\n",
 		deviceid_fmt(&dsaddr->dev_id));
-	print_multipath_list(dsaddr);
+	print_ds_list(dsaddr);
 
 	hash = _deviceid_hash(&dsaddr->dev_id);
 	hlist_add_head(&dsaddr->hash_node, &hlist->dev_list[hash]);
@@ -306,34 +294,29 @@ static void
 nfs4_pnfs_device_destroy(struct nfs4_file_layout_dsaddr *dsaddr,
 			 struct nfs4_pnfs_dev_hlist *hlist)
 {
-	struct nfs4_multipath *multipath;
 	struct nfs4_pnfs_ds *ds;
 	HLIST_HEAD(release);
-	int i, j;
+	int i;
 
 	if (!dsaddr)
 		return;
 
 	dprintk("%s: dev_id=%s\ndev_list:\n", __func__,
 		deviceid_fmt(&dsaddr->dev_id));
-	print_multipath_list(dsaddr);
+	print_ds_list(dsaddr);
 
 	write_lock(&hlist->dev_lock);
 	hlist_del_init(&dsaddr->hash_node);
 
-	multipath = &dsaddr->multipath_list[0];
-	for (i = 0; i < dsaddr->multipath_count; i++) {
-		for (j = 0; j < multipath->num_ds; j++) {
-			ds = multipath->ds_list[j];
-			if (ds != NULL) {
-				/* if we are last user - move to release list */
-				if (atomic_dec_and_test(&ds->ds_count)) {
-					hlist_del_init(&ds->ds_node);
-					hlist_add_head(&ds->ds_node, &release);
-				}
+	for (i = 0; i < dsaddr->ds_num; i++) {
+		ds = dsaddr->ds_list[i];
+		if (ds != NULL) {
+			/* if we are last user - move to release list */
+			if (atomic_dec_and_test(&ds->ds_count)) {
+				hlist_del_init(&ds->ds_node);
+				hlist_add_head(&ds->ds_node, &release);
 			}
 		}
-		multipath++;
 	}
 	write_unlock(&hlist->dev_lock);
 	while (!hlist_empty(&release)) {
@@ -341,8 +324,6 @@ nfs4_pnfs_device_destroy(struct nfs4_file_layout_dsaddr *dsaddr,
 		hlist_del(&ds->ds_node);
 		destroy_ds(ds);
 	}
-	kfree(dsaddr->multipath_list);
-	kfree(dsaddr->stripe_indices);
 	kfree(dsaddr);
 }
 
@@ -540,47 +521,44 @@ static struct nfs4_file_layout_dsaddr*
 decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 {
 	int i, dummy;
+	u32 cnt, num;
 	u8 *indexp;
 	uint32_t *p = (u32 *)pdev->area, *indicesp;
 	struct nfs4_file_layout_dsaddr *dsaddr;
-	struct nfs4_multipath *multipath;
-
-	dsaddr = kzalloc(sizeof(*dsaddr), GFP_KERNEL);
-	if (!dsaddr)
-		goto out_err;
 
 	/* Get the stripe count (number of stripe index) */
-	READ32(dsaddr->stripe_count);
-	dprintk("%s stripe count  %d\n", __func__, dsaddr->stripe_count);
-	if (dsaddr->stripe_count > NFS4_PNFS_MAX_STRIPE_CNT) {
+	READ32(cnt);
+	dprintk("%s stripe count  %d\n", __func__, cnt);
+	if (cnt > NFS4_PNFS_MAX_STRIPE_CNT) {
 		printk(KERN_WARNING "%s: stripe count %d greater than "
 		       "supported maximum %d\n", __func__,
-			dsaddr->stripe_count, NFS4_PNFS_MAX_STRIPE_CNT);
-		dsaddr->stripe_count = 0;
-		goto out_err_free;
+			cnt, NFS4_PNFS_MAX_STRIPE_CNT);
+		goto out_err;
 	}
 
 	/* Check the multipath list count */
 	indicesp = p;
-	p += XDR_QUADLEN(dsaddr->stripe_count << 2);
-	READ32(dsaddr->multipath_count);
-	dprintk("%s multipath_count %u\n", __func__, dsaddr->multipath_count);
-	if (dsaddr->multipath_count > NFS4_PNFS_MAX_MULTI_CNT) {
+	p += XDR_QUADLEN(cnt << 2);
+	READ32(num);
+	dprintk("%s ds_num %u\n", __func__, num);
+	if (num > NFS4_PNFS_MAX_MULTI_CNT) {
 		printk(KERN_WARNING "%s: multipath count %d greater than "
 			"supported maximum %d\n", __func__,
-			dsaddr->multipath_count, NFS4_PNFS_MAX_MULTI_CNT);
-		dsaddr->multipath_count = 0;
-		goto out_err_free;
+			num, NFS4_PNFS_MAX_MULTI_CNT);
+		goto out_err;
 	}
-	dsaddr->stripe_indices = kzalloc(sizeof(u8) * dsaddr->stripe_count,
-					 GFP_KERNEL);
+	dsaddr = kzalloc(sizeof(*dsaddr) +
+			(sizeof(struct nfs4_pnfs_ds *) * (num - 1)),
+			GFP_KERNEL);
+	if (!dsaddr)
+		goto out_err;
+
+	dsaddr->stripe_indices = kzalloc(sizeof(u8) * cnt, GFP_KERNEL);
 	if (!dsaddr->stripe_indices)
 		goto out_err_free;
 
-	dsaddr->multipath_list = kzalloc(sizeof(struct nfs4_multipath) *
-					 dsaddr->multipath_count, GFP_KERNEL);
-	if (!dsaddr->multipath_list)
-		goto out_err_free;
+	dsaddr->stripe_count = cnt;
+	dsaddr->ds_num = num;
 
 	memcpy(&dsaddr->dev_id, &pdev->dev_id, NFS4_PNFS_DEVICEID4_SIZE);
 
@@ -595,19 +573,22 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	/* Skip already read multipath list count */
 	p++;
 
-	multipath = &dsaddr->multipath_list[0];
-	for (i = 0; i < dsaddr->multipath_count; i++) {
+	for (i = 0; i < dsaddr->ds_num; i++) {
 		int j;
 
-		READ32(multipath->num_ds);
-		if (multipath->num_ds > NFS4_PNFS_MAX_MULTI_DS) {
+		READ32(dummy); /* multipath count */
+		if (dummy > 1) {
 			printk(KERN_WARNING
 			       "%s: Multipath count %d not supported, "
-			       "skipping all greater than %d\n", __func__,
-				multipath->num_ds, NFS4_PNFS_MAX_MULTI_DS);
+			       "skipping all greater than 1\n", __func__,
+				dummy);
 		}
-		for (j = 0; j < multipath->num_ds; j++) {
-			if (j >= NFS4_PNFS_MAX_MULTI_DS) {
+		for (j = 0; j < dummy; j++) {
+			if (j == 0) {
+				dsaddr->ds_list[i] = decode_and_add_ds(&p, mt);
+				if (dsaddr->ds_list[i] == NULL)
+					goto out_err_free;
+			} else {
 				u32 len;
 				/* skip extra multipath */
 				READ32(len);
@@ -616,11 +597,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 				p += XDR_QUADLEN(len);
 				continue;
 			}
-			multipath->ds_list[j] = decode_and_add_ds(&p, mt);
-			if (multipath->ds_list[j] == NULL)
-				goto out_err_free;
 		}
-		multipath++;
 	}
 	return dsaddr;
 
@@ -771,7 +748,7 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	struct inode *inode = PNFS_INODE(lseg->layout);
 	struct nfs4_file_layout_dsaddr *dsaddr;
 	u64 tmp, tmp2;
-	u32 stripe_idx, end_idx, multipath_idx;
+	u32 stripe_idx, end_idx, ds_idx;
 
 	if (!layout)
 		return 1;
@@ -798,10 +775,10 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	BUG_ON(end_idx != stripe_idx);
 	BUG_ON(stripe_idx >= dsaddr->stripe_count);
 
-	multipath_idx = dsaddr->stripe_indices[stripe_idx];
-	dserver->multipath = &dsaddr->multipath_list[multipath_idx];
+	ds_idx = dsaddr->stripe_indices[stripe_idx];
+	dserver->ds = dsaddr->ds_list[ds_idx];
 
-	if (dserver->multipath == NULL) {
+	if (dserver->ds == NULL) {
 		printk(KERN_ERR "%s: No data server for device id (%s)!! \n",
 			__func__, deviceid_fmt(&layout->dev_id));
 		return 1;
@@ -809,13 +786,13 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	if (layout->num_fh == 1)
 		dserver->fh = &layout->fh_array[0];
 	else
-		dserver->fh = &layout->fh_array[multipath_idx];
+		dserver->fh = &layout->fh_array[ds_idx];
 
-	dprintk("%s: dev_id=%s, ip:port=%s, multipath_idx=%u stripe_idx=%u, "
+	dprintk("%s: dev_id=%s, ip:port=%s, ds_idx=%u stripe_idx=%u, "
 		"offset=%llu, count=%Zu\n",
 		__func__, deviceid_fmt(&layout->dev_id),
-		dserver->multipath->ds_list[0]->r_addr,
-		multipath_idx, stripe_idx, offset, count);
+		dserver->ds->r_addr,
+		ds_idx, stripe_idx, offset, count);
 
 	return 0;
 }
-- 
1.6.6

[PATCH 04/14] SQUASHME pnfs_submit: wait for I/O for data server connect

[andros]

From: Andy Adamson <andros@netapp.com>

Don't create (and maintain) a session to a data server until needed.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |   34 ++++++++++++++++------------------
 1 files changed, 16 insertions(+), 18 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index ea25bd2..cda0bce 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -458,10 +458,9 @@ nfs4_pnfs_ds_add(struct filelayout_mount_type *mt, struct nfs4_pnfs_ds **dsp,
 static struct nfs4_pnfs_ds *
 decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 {
-	struct nfs_server *mds_srv = NFS_SB(mt->fl_sb);
 	struct nfs4_pnfs_ds *ds = NULL;
 	char r_addr[29]; /* max size of ip/port string */
-	int len, err;
+	int len;
 	u32 ip_addr, port;
 	int tmp[6];
 	uint32_t *p = *pp;
@@ -495,19 +494,6 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 
 	nfs4_pnfs_ds_add(mt, &ds, ip_addr, port, r_addr, len);
 
-	/* XXX: Don't connect to data servers here, because we
-	 * don't want any un-used (never used!) connections.
-	 * We should wait until I/O demands use of the data server.
-	 */
-	if (!ds->ds_clp) {
-		err = nfs4_pnfs_ds_create(mds_srv, ds);
-		if (err) {
-			printk(KERN_ERR "%s nfs4_pnfs_ds_create error %d\n",
-			       __func__, err);
-			goto out_err;
-		}
-	}
-
 	dprintk("%s: addr:port string = %s\n", __func__, r_addr);
 	return ds;
 out_err:
@@ -746,6 +732,7 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 {
 	struct nfs4_filelayout_segment *layout = LSEG_LD_DATA(lseg);
 	struct inode *inode = PNFS_INODE(lseg->layout);
+	struct nfs_server *mds_srv = NFS_SERVER(inode);
 	struct nfs4_file_layout_dsaddr *dsaddr;
 	u64 tmp, tmp2;
 	u32 stripe_idx, end_idx, ds_idx;
@@ -776,13 +763,24 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 	BUG_ON(stripe_idx >= dsaddr->stripe_count);
 
 	ds_idx = dsaddr->stripe_indices[stripe_idx];
-	dserver->ds = dsaddr->ds_list[ds_idx];
-
-	if (dserver->ds == NULL) {
+	if (dsaddr->ds_list[ds_idx] == NULL) {
 		printk(KERN_ERR "%s: No data server for device id (%s)!! \n",
 			__func__, deviceid_fmt(&layout->dev_id));
 		return 1;
 	}
+
+	if (!dsaddr->ds_list[ds_idx]->ds_clp) {
+		int err;
+
+		err = nfs4_pnfs_ds_create(mds_srv, dsaddr->ds_list[ds_idx]);
+		if (err) {
+			printk(KERN_ERR "%s nfs4_pnfs_ds_create error %d\n",
+			       __func__, err);
+			return 1;
+		}
+	}
+	dserver->ds = dsaddr->ds_list[ds_idx];
+
 	if (layout->num_fh == 1)
 		dserver->fh = &layout->fh_array[0];
 	else
-- 
1.6.6

[PATCH 05/14] SQUASHME pnfs_submit: remove xdr macros

[andros]

From: Andy Adamson <andros@netapp.com>

As per the rest of the nfs client, remove the old xdr macros

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.c    |   16 +++++++++-------
 fs/nfs/nfs4filelayout.h    |   11 -----------
 fs/nfs/nfs4filelayoutdev.c |   25 +++++++++++++------------
 3 files changed, 22 insertions(+), 30 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
index abdf691..20b3ae2 100644
--- a/fs/nfs/nfs4filelayout.c
+++ b/fs/nfs/nfs4filelayout.c
@@ -459,8 +459,9 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 
 	dprintk("%s: set_layout_map Begin\n", __func__);
 
-	COPYMEM(&fl->dev_id, NFS4_PNFS_DEVICEID4_SIZE);
-	READ32(nfl_util);
+	memcpy(&fl->dev_id, p, NFS4_PNFS_DEVICEID4_SIZE);
+	p += XDR_QUADLEN(NFS4_PNFS_DEVICEID4_SIZE);
+	nfl_util = be32_to_cpup(p++);
 	if (nfl_util & NFL4_UFLG_COMMIT_THRU_MDS)
 		fl->commit_through_mds = 1;
 	if (nfl_util & NFL4_UFLG_DENSE)
@@ -477,9 +478,9 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 		flo->stripe_unit = fl->stripe_unit;
 	}
 
-	READ32(fl->first_stripe_index);
-	READ64(fl->pattern_offset);
-	READ32(fl->num_fh);
+	fl->first_stripe_index = be32_to_cpup(p++);
+	p = xdr_decode_hyper(p, &fl->pattern_offset);
+	fl->num_fh = be32_to_cpup(p++);
 
 	dprintk("%s: nfl_util 0x%X num_fh %u fsi %u po %llu dev_id %s\n",
 		__func__, nfl_util, fl->num_fh, fl->first_stripe_index,
@@ -499,7 +500,7 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 
 	for (i = 0; i < fl->num_fh; i++) {
 		/* fh */
-		READ32(fl->fh_array[i].size);
+		fl->fh_array[i].size = be32_to_cpup(p++);
 		if (sizeof(struct nfs_fh) < fl->fh_array[i].size) {
 			printk(KERN_ERR "Too big fh %d received %d\n",
 				i, fl->fh_array[i].size);
@@ -509,7 +510,8 @@ filelayout_set_layout(struct nfs4_filelayout *flo,
 			fl->num_fh = 0;
 			break;
 		}
-		COPYMEM(fl->fh_array[i].data, fl->fh_array[i].size);
+		memcpy(fl->fh_array[i].data, p, fl->fh_array[i].size);
+		p += XDR_QUADLEN(fl->fh_array[i].size);
 		dprintk("DEBUG: %s: fh len %d\n", __func__,
 					fl->fh_array[i].size);
 	}
diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index 0df1eaf..04cca7e 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -105,17 +105,6 @@ int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 u32 filelayout_dserver_get_index(loff_t offset,
 				 struct nfs4_file_layout_dsaddr *di,
 				 struct nfs4_filelayout_segment *layout);
-
-#define READ32(x)         (x) = ntohl(*p++)
-#define READ64(x)         do {			\
-	(x) = (u64)ntohl(*p++) << 32;		\
-	(x) |= ntohl(*p++);			\
-} while (0)
-#define COPYMEM(x,nbytes) do {			\
-	memcpy((x), p, nbytes);			\
-	p += XDR_QUADLEN(nbytes);		\
-} while (0)
-
 struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id);
diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index cda0bce..99a7ceb 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -88,8 +88,8 @@ deviceid_fmt(const struct pnfs_deviceid *dev_id)
 	uint32_t *p = (uint32_t *)dev_id->data;
 	uint64_t major, minor;
 
-	READ64(major);
-	READ64(minor);
+	p = xdr_decode_hyper(p, &major);
+	p = xdr_decode_hyper(p, &minor);
 
 	sprintf(buf, "%08llu %08llu", major, minor);
 	return buf;
@@ -467,7 +467,7 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 
 	dprintk("%s enter\n", __func__);
 	/* check and skip r_netid */
-	READ32(len);
+	len = be32_to_cpup(p++);
 	/* "tcp" */
 	if (len != 3) {
 		printk("%s: ERROR: non TCP r_netid len %d\n",
@@ -476,15 +476,16 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 	}
 	/* Read the bytes into a temporary buffer */
 	/* XXX: should probably sanity check them */
-	READ32(tmp[0]);
+	tmp[0] = be32_to_cpup(p++);
 
-	READ32(len);
+	len = be32_to_cpup(p++);
 	if (len > 29) {
 		printk("%s: ERROR: Device ip/port too long (%d)\n",
 			__func__, len);
 		goto out_err;
 	}
-	COPYMEM(r_addr, len);
+	memcpy(r_addr, p, len);
+	p += XDR_QUADLEN(len);
 	*pp = p;
 	r_addr[len] = '\0';
 	sscanf(r_addr, "%d.%d.%d.%d.%d.%d", &tmp[0], &tmp[1],
@@ -513,7 +514,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	struct nfs4_file_layout_dsaddr *dsaddr;
 
 	/* Get the stripe count (number of stripe index) */
-	READ32(cnt);
+	cnt = be32_to_cpup(p++);
 	dprintk("%s stripe count  %d\n", __func__, cnt);
 	if (cnt > NFS4_PNFS_MAX_STRIPE_CNT) {
 		printk(KERN_WARNING "%s: stripe count %d greater than "
@@ -525,7 +526,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	/* Check the multipath list count */
 	indicesp = p;
 	p += XDR_QUADLEN(cnt << 2);
-	READ32(num);
+	num = be32_to_cpup(p++);
 	dprintk("%s ds_num %u\n", __func__, num);
 	if (num > NFS4_PNFS_MAX_MULTI_CNT) {
 		printk(KERN_WARNING "%s: multipath count %d greater than "
@@ -552,7 +553,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	p = indicesp;
 	indexp = &dsaddr->stripe_indices[0];
 	for (i = 0; i < dsaddr->stripe_count; i++) {
-		READ32(dummy);
+		dummy = be32_to_cpup(p++);
 		*indexp = dummy; /* bound by NFS4_PNFS_MAX_MULTI_CNT */
 		indexp++;
 	}
@@ -562,7 +563,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	for (i = 0; i < dsaddr->ds_num; i++) {
 		int j;
 
-		READ32(dummy); /* multipath count */
+		dummy = be32_to_cpup(p++); /* multipath count */
 		if (dummy > 1) {
 			printk(KERN_WARNING
 			       "%s: Multipath count %d not supported, "
@@ -577,9 +578,9 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 			} else {
 				u32 len;
 				/* skip extra multipath */
-				READ32(len);
+				len = be32_to_cpup(p++);
 				p += XDR_QUADLEN(len);
-				READ32(len);
+				len = be32_to_cpup(p++);
 				p += XDR_QUADLEN(len);
 				continue;
 			}
-- 
1.6.6

[PATCH 06/14] SQUASHME pnfs_submit: remove unused nfs4_pnfs_device_put

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |   21 ---------------------
 1 files changed, 0 insertions(+), 21 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 99a7ceb..db6817d 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -795,24 +795,3 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 
 	return 0;
 }
-
-/* Currently not used.
- * I have disabled checking the device count until we can think of a good way
- * to call nfs4_pnfs_device_put in a generic way from the pNFS client.
- * The only way I think think of is to put the nfs4_file_layout_dsaddr directly
- * in the nfs4_write/read_data structure, which breaks the clear line between
- * the pNFS client and layout drivers.  If I did do this, then I could call
- * an ioctl on the NFSv4 file layout driver to decrement the device count.
- */
-#if 0
-static void
-nfs4_pnfs_device_put(struct nfs_server *server,
-		     struct nfs4_pnfs_dev_hlist *hlist,
-		     struct nfs4_file_layout_dsaddr *dsaddr)
-{
-	dprintk("nfs4_pnfs_device_put: dev_id=%u\n", dsaddr->dev_id);
-	/* XXX Do we need to invoke this put_client? */
-	/* server->rpc_ops->put_client(dsaddr->clp); */
-	atomic_dec(&dsaddr->count);
-}
-#endif
-- 
1.6.6

[PATCH 07/14] SQUASHME pnfs_submit: change get_device_info parameter

[andros]

From: Andy Adamson <andros@netapp.com>

In preparation to use the generic deviceid cache.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.c    |    2 +-
 fs/nfs/nfs4filelayout.h    |    2 +-
 fs/nfs/nfs4filelayoutdev.c |    9 ++++-----
 3 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
index 20b3ae2..f222c3b 100644
--- a/fs/nfs/nfs4filelayout.c
+++ b/fs/nfs/nfs4filelayout.c
@@ -399,7 +399,7 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
 	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
 					     &fl->dev_id);
 	if (dsaddr == NULL) {
-		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
+		dsaddr = get_device_info(PNFS_INODE(lo), &fl->dev_id);
 		if (dsaddr == NULL) {
 			dprintk("%s NO device for dev_id %s\n",
 				__func__, deviceid_fmt(&fl->dev_id));
diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index 04cca7e..560f7f6 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -109,6 +109,6 @@ struct nfs4_file_layout_dsaddr *
 nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
 			   struct pnfs_deviceid *dev_id);
 struct nfs4_file_layout_dsaddr *
-get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
+get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id);
 
 #endif /* FS_NFS_NFS4FILELAYOUT_H */
diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index db6817d..9844a9c 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -623,8 +623,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
  * of available devices, and return it.
  */
 struct nfs4_file_layout_dsaddr *
-get_device_info(struct filelayout_mount_type *mt,
-		struct pnfs_deviceid *dev_id)
+get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id)
 {
 	struct pnfs_device *pdev = NULL;
 	int maxpages = NFS4_GETDEVINFO_MAXSIZE >> PAGE_SHIFT;
@@ -632,7 +631,7 @@ get_device_info(struct filelayout_mount_type *mt,
 	struct nfs4_file_layout_dsaddr *dsaddr = NULL;
 	int rc, i, j, minpages = 1;
 
-	dprintk("%s mt %p\n", __func__, mt);
+	dprintk("%s \n", __func__);
 	pdev = kzalloc(sizeof(struct pnfs_device), GFP_KERNEL);
 	if (pdev == NULL)
 		return NULL;
@@ -664,7 +663,7 @@ retry_once:
 	/* TODO: Update types when CB_NOTIFY_DEVICEID is available */
 	pdev->dev_notify_types = 0;
 
-	rc = pnfs_callback_ops->nfs_getdeviceinfo(mt->fl_sb, pdev);
+	rc = pnfs_callback_ops->nfs_getdeviceinfo(inode->i_sb, pdev);
 	/* Retry once with the returned mincount if a page was too small */
 	dprintk("%s getdevice info returns %d minpages %d\n", __func__, rc,
 		minpages);
@@ -682,7 +681,7 @@ retry_once:
 	/* Found new device, need to decode it and then add it to the
 	 * list of known devices for this mountpoint.
 	 */
-	dsaddr = decode_and_add_device(mt, pdev);
+	dsaddr = decode_and_add_device(FILE_MT(inode), pdev);
 out_free:
 	if (minpages > 1 && pdev->area != NULL)
 		vunmap(pdev->area);
-- 
1.6.6

[PATCH 08/14] SQUASHME pnfs_submit: change decode_and_add_device parameter

[andros]

From: Andy Adamson <andros@netapp.com>

In preparation to use the generic deviceid cache.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 9844a9c..bb8a84a 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -601,19 +601,19 @@ out_err:
  * Must at some point be followed up with nfs4_pnfs_device_destroy
  */
 static struct nfs4_file_layout_dsaddr*
-decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
+decode_and_add_device(struct inode *inode, struct pnfs_device *dev)
 {
 	struct nfs4_file_layout_dsaddr *dsaddr;
 
-	dsaddr = decode_device(mt, dev);
+	dsaddr = decode_device(FILE_MT(inode), dev);
 	if (!dsaddr) {
 		printk(KERN_WARNING "%s: Could not decode device\n",
 			__func__);
-		nfs4_pnfs_device_destroy(dsaddr, mt->hlist);
+		nfs4_pnfs_device_destroy(dsaddr, FILE_MT(inode)->hlist);
 		return NULL;
 	}
 
-	if (nfs4_pnfs_device_add(mt, dsaddr))
+	if (nfs4_pnfs_device_add(FILE_MT(inode), dsaddr))
 		return NULL;
 
 	return dsaddr;
@@ -681,7 +681,7 @@ retry_once:
 	/* Found new device, need to decode it and then add it to the
 	 * list of known devices for this mountpoint.
 	 */
-	dsaddr = decode_and_add_device(FILE_MT(inode), pdev);
+	dsaddr = decode_and_add_device(inode, pdev);
 out_free:
 	if (minpages > 1 && pdev->area != NULL)
 		vunmap(pdev->area);
-- 
1.6.6

[PATCH 09/14] SQUASHME pnfs_submit: change decode_device parameter

[andros]

From: Andy Adamson <andros@netapp.com>

In preparation to use the generic deviceid cache

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index bb8a84a..506ede1 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -505,7 +505,7 @@ out_err:
 /* Decode opaque device data and return the result
  */
 static struct nfs4_file_layout_dsaddr*
-decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
+decode_device(struct inode *ino, struct pnfs_device *pdev)
 {
 	int i, dummy;
 	u32 cnt, num;
@@ -572,7 +572,8 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 		}
 		for (j = 0; j < dummy; j++) {
 			if (j == 0) {
-				dsaddr->ds_list[i] = decode_and_add_ds(&p, mt);
+				dsaddr->ds_list[i] = decode_and_add_ds(&p,
+								FILE_MT(ino));
 				if (dsaddr->ds_list[i] == NULL)
 					goto out_err_free;
 			} else {
@@ -589,7 +590,7 @@ decode_device(struct filelayout_mount_type *mt, struct pnfs_device *pdev)
 	return dsaddr;
 
 out_err_free:
-	nfs4_pnfs_device_destroy(dsaddr, mt->hlist);
+	nfs4_pnfs_device_destroy(dsaddr, FILE_MT(ino)->hlist);
 out_err:
 	dprintk("%s ERROR: returning NULL\n", __func__);
 	return NULL;
@@ -605,7 +606,7 @@ decode_and_add_device(struct inode *inode, struct pnfs_device *dev)
 {
 	struct nfs4_file_layout_dsaddr *dsaddr;
 
-	dsaddr = decode_device(FILE_MT(inode), dev);
+	dsaddr = decode_device(inode, dev);
 	if (!dsaddr) {
 		printk(KERN_WARNING "%s: Could not decode device\n",
 			__func__);
-- 
1.6.6

[PATCH 10/14] SQUASHME pnfs_submit: change decode_and_add_ds parameter

[andros]

From: Andy Adamson <andros@netapp.com>

In preparation to use the generic deviceid cache

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |    7 +++----
 1 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 506ede1..4882149 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -456,7 +456,7 @@ nfs4_pnfs_ds_add(struct filelayout_mount_type *mt, struct nfs4_pnfs_ds **dsp,
 }
 
 static struct nfs4_pnfs_ds *
-decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
+decode_and_add_ds(uint32_t **pp, struct inode *inode)
 {
 	struct nfs4_pnfs_ds *ds = NULL;
 	char r_addr[29]; /* max size of ip/port string */
@@ -493,7 +493,7 @@ decode_and_add_ds(uint32_t **pp, struct filelayout_mount_type *mt)
 	ip_addr = htonl((tmp[0]<<24) | (tmp[1]<<16) | (tmp[2]<<8) | (tmp[3]));
 	port = htons((tmp[4] << 8) | (tmp[5]));
 
-	nfs4_pnfs_ds_add(mt, &ds, ip_addr, port, r_addr, len);
+	nfs4_pnfs_ds_add(FILE_MT(inode), &ds, ip_addr, port, r_addr, len);
 
 	dprintk("%s: addr:port string = %s\n", __func__, r_addr);
 	return ds;
@@ -572,8 +572,7 @@ decode_device(struct inode *ino, struct pnfs_device *pdev)
 		}
 		for (j = 0; j < dummy; j++) {
 			if (j == 0) {
-				dsaddr->ds_list[i] = decode_and_add_ds(&p,
-								FILE_MT(ino));
+				dsaddr->ds_list[i] = decode_and_add_ds(&p,ino);
 				if (dsaddr->ds_list[i] == NULL)
 					goto out_err_free;
 			} else {
-- 
1.6.6

[PATCH 11/14] SQUASHME pnfs_submit: change nfs4_pnfs_ds_add parameter

[andros]

From: Andy Adamson <andros@netapp.com>

In preparation to use the generic deviceid cache

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 4882149..2cbe931 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -416,11 +416,11 @@ nfs4_pnfs_device_add(struct filelayout_mount_type *mt,
 }
 
 static void
-nfs4_pnfs_ds_add(struct filelayout_mount_type *mt, struct nfs4_pnfs_ds **dsp,
+nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 		 u32 ip_addr, u32 port, char *r_addr, int len)
 {
 	struct nfs4_pnfs_ds *tmp_ds, *ds;
-	struct nfs4_pnfs_dev_hlist *hlist = mt->hlist;
+	struct nfs4_pnfs_dev_hlist *hlist = FILE_MT(inode)->hlist;
 
 	*dsp = NULL;
 
@@ -493,7 +493,7 @@ decode_and_add_ds(uint32_t **pp, struct inode *inode)
 	ip_addr = htonl((tmp[0]<<24) | (tmp[1]<<16) | (tmp[2]<<8) | (tmp[3]));
 	port = htons((tmp[4] << 8) | (tmp[5]));
 
-	nfs4_pnfs_ds_add(FILE_MT(inode), &ds, ip_addr, port, r_addr, len);
+	nfs4_pnfs_ds_add(inode, &ds, ip_addr, port, r_addr, len);
 
 	dprintk("%s: addr:port string = %s\n", __func__, r_addr);
 	return ds;
-- 
1.6.6

[PATCH 12/14] SQUASHME pnfs_submit: remove filelayout CB_NOTIFY_DEVICE support

[andros]

From: Andy Adamson <andros@netapp.com>

We won't support CB_NOTIFY_DEVICE in the first submission stage.
Remove filelayout_device_delete and associated functionality.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.c    |   16 ----------------
 fs/nfs/nfs4filelayout.h    |    2 --
 fs/nfs/nfs4filelayoutdev.c |   18 ------------------
 3 files changed, 0 insertions(+), 36 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
index f222c3b..0530b59 100644
--- a/fs/nfs/nfs4filelayout.c
+++ b/fs/nfs/nfs4filelayout.c
@@ -135,21 +135,6 @@ filelayout_uninitialize_mountpoint(struct pnfs_mount_type *mountid)
 	return 0;
 }
 
-int
-filelayout_device_delete(struct pnfs_mount_type *mountid,
-				struct pnfs_deviceid *dev_id)
-{
-	struct filelayout_mount_type *fl_mt = NULL;
-
-	if (mountid) {
-		fl_mt = (struct filelayout_mount_type *)mountid->mountid;
-
-		if (fl_mt != NULL)
-			nfs4_pnfs_dev_destroy(fl_mt->hlist, dev_id);
-	}
-	return 0;
-}
-
 /* This function is used by the layout driver to calculate the
  * offset of the file on the dserver based on whether the
  * layout type is STRIPE_DENSE or STRIPE_SPARSE
@@ -797,7 +782,6 @@ struct layoutdriver_io_operations filelayout_io_operations = {
 	.free_lseg               = filelayout_free_lseg,
 	.initialize_mountpoint   = filelayout_initialize_mountpoint,
 	.uninitialize_mountpoint = filelayout_uninitialize_mountpoint,
-	.device_delete           = filelayout_device_delete,
 };
 
 struct layoutdriver_policy_operations filelayout_policy_operations = {
diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index 560f7f6..e988891 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -96,8 +96,6 @@ extern void print_ds(struct nfs4_pnfs_ds *ds);
 char *deviceid_fmt(const struct pnfs_deviceid *dev_id);
 int  nfs4_pnfs_devlist_init(struct nfs4_pnfs_dev_hlist *hlist);
 void nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist);
-void nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
-				struct pnfs_deviceid *dev_id);
 int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
 			  loff_t offset,
 			  size_t count,
diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 2cbe931..2d98ca3 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -368,24 +368,6 @@ nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist)
 	}
 }
 
-/* De-alloc a devices for a mount point. */
-void
-nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
-			struct pnfs_deviceid *dev_id)
-{
-	struct nfs4_file_layout_dsaddr *dsaddr;
-
-	if (hlist == NULL)
-		return;
-
-	dprintk("%s: dev_id=%s\n", __func__, deviceid_fmt(dev_id));
-
-	dsaddr = nfs4_pnfs_device_item_find(hlist, dev_id);
-	if (dsaddr)
-		/* nfs4_pnfs_device_destroy grabs hlist->dev_lock */
-		nfs4_pnfs_device_destroy(dsaddr, hlist);
-}
-
 /*
  * Add the device to the list of available devices for this mount point.
  * The * rpc client is created during first I/O.
-- 
1.6.6

[PATCH 13/14] SQUASHME pnfs_submit: stand alone data server cache

[andros]

From: Andy Adamson <andros@netapp.com>

Data servers can be associated with any device ID from any server.
Multiple file layout meta data servers can share a data server.
Implement a stand alone data server cache to share data servers and
their session between deviceid's and meta data servers.

The data server cache is only changed at GETDEVICEINFO return or umount.
Once in the cache and referenced by a deviceid structure, the use of the
data server in the I/O path is via deviceid struct array indexes.
Therefore, nether an hlist nor rwspin locks are needed.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayout.h    |    3 +-
 fs/nfs/nfs4filelayoutdev.c |   57 ++++++++++++++++---------------------------
 2 files changed, 22 insertions(+), 38 deletions(-)

diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
index e988891..809747f 100644
--- a/fs/nfs/nfs4filelayout.h
+++ b/fs/nfs/nfs4filelayout.h
@@ -34,7 +34,7 @@ enum stripetype4 {
 
 /* Individual ip address */
 struct nfs4_pnfs_ds {
-	struct hlist_node 	ds_node;  /* nfs4_pnfs_dev_hlist dev_dslist */
+	struct list_head 	ds_node;  /* nfs4_pnfs_dev_hlist dev_dslist */
 	u32 			ds_ip_addr;
 	u32 			ds_port;
 	struct nfs_client	*ds_clp;
@@ -54,7 +54,6 @@ struct nfs4_file_layout_dsaddr {
 struct nfs4_pnfs_dev_hlist {
 	rwlock_t		dev_lock;
 	struct hlist_head	dev_list[NFS4_PNFS_DEV_HASH_SIZE];
-	struct hlist_head	dev_dslist[NFS4_PNFS_DEV_HASH_SIZE];
 };
 
 /*
diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 2d98ca3..50e2865 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -52,6 +52,9 @@
 
 #define NFSDBG_FACILITY		NFSDBG_PNFS_LD
 
+DEFINE_SPINLOCK(nfs4_ds_cache_lock);
+static LIST_HEAD(nfs4_data_server_cache);
+
 void
 print_ds(struct nfs4_pnfs_ds *ds)
 {
@@ -131,21 +134,16 @@ _device_lookup(struct nfs4_pnfs_dev_hlist *hlist,
 	return NULL;
 }
 
-/* Assumes lock is held */
+/* nfs4_ds_cache_lock is held */
 static inline struct nfs4_pnfs_ds *
-_data_server_lookup(struct nfs4_pnfs_dev_hlist *hlist, u32 ip_addr, u32 port)
+_data_server_lookup(u32 ip_addr, u32 port)
 {
-	unsigned long      hash;
-	struct hlist_node *np;
+	struct nfs4_pnfs_ds *ds;
 
 	dprintk("_data_server_lookup: ip_addr=%x port=%hu\n",
 			ntohl(ip_addr), ntohs(port));
 
-	hash = hash_long(ip_addr, NFS4_PNFS_DEV_HASH_BITS);
-
-	hlist_for_each(np, &hlist->dev_dslist[hash]) {
-		struct nfs4_pnfs_ds *ds;
-		ds = hlist_entry(np, struct nfs4_pnfs_ds, ds_node);
+	list_for_each_entry(ds, &nfs4_data_server_cache, ds_node) {
 		if (ds->ds_ip_addr == ip_addr &&
 		    ds->ds_port == port) {
 			return ds;
@@ -170,19 +168,6 @@ _device_add(struct nfs4_pnfs_dev_hlist *hlist,
 	hlist_add_head(&dsaddr->hash_node, &hlist->dev_list[hash]);
 }
 
-/* Assumes lock is held */
-static inline void
-_data_server_add(struct nfs4_pnfs_dev_hlist *hlist, struct nfs4_pnfs_ds *ds)
-{
-	unsigned long      hash;
-
-	dprintk("_data_server_add: ip_addr=%x port=%hu\n",
-			ntohl(ds->ds_ip_addr), ntohs(ds->ds_port));
-
-	hash = hash_long(ds->ds_ip_addr, NFS4_PNFS_DEV_HASH_BITS);
-	hlist_add_head(&ds->ds_node, &hlist->dev_dslist[hash]);
-}
-
 /* Create an rpc to the data server defined in 'dev_list' */
 static int
 nfs4_pnfs_ds_create(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds)
@@ -295,7 +280,7 @@ nfs4_pnfs_device_destroy(struct nfs4_file_layout_dsaddr *dsaddr,
 			 struct nfs4_pnfs_dev_hlist *hlist)
 {
 	struct nfs4_pnfs_ds *ds;
-	HLIST_HEAD(release);
+	LIST_HEAD(release);
 	int i;
 
 	if (!dsaddr)
@@ -312,16 +297,18 @@ nfs4_pnfs_device_destroy(struct nfs4_file_layout_dsaddr *dsaddr,
 		ds = dsaddr->ds_list[i];
 		if (ds != NULL) {
 			/* if we are last user - move to release list */
-			if (atomic_dec_and_test(&ds->ds_count)) {
-				hlist_del_init(&ds->ds_node);
-				hlist_add_head(&ds->ds_node, &release);
+			if (atomic_dec_and_lock(&ds->ds_count,
+						&nfs4_ds_cache_lock)) {
+				list_del_init(&ds->ds_node);
+				spin_unlock(&nfs4_ds_cache_lock);
+				list_add(&ds->ds_node, &release);
 			}
 		}
 	}
 	write_unlock(&hlist->dev_lock);
-	while (!hlist_empty(&release)) {
-		ds = hlist_entry(release.first, struct nfs4_pnfs_ds, ds_node);
-		hlist_del(&ds->ds_node);
+	while (!list_empty(&release)) {
+		ds = list_entry(release.next, struct nfs4_pnfs_ds, ds_node);
+		list_del(&ds->ds_node);
 		destroy_ds(ds);
 	}
 	kfree(dsaddr);
@@ -336,7 +323,6 @@ nfs4_pnfs_devlist_init(struct nfs4_pnfs_dev_hlist *hlist)
 
 	for (i = 0; i < NFS4_PNFS_DEV_HASH_SIZE; i++) {
 		INIT_HLIST_HEAD(&hlist->dev_list[i]);
-		INIT_HLIST_HEAD(&hlist->dev_dslist[i]);
 	}
 
 	return 0;
@@ -402,7 +388,6 @@ nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 		 u32 ip_addr, u32 port, char *r_addr, int len)
 {
 	struct nfs4_pnfs_ds *tmp_ds, *ds;
-	struct nfs4_pnfs_dev_hlist *hlist = FILE_MT(inode)->hlist;
 
 	*dsp = NULL;
 
@@ -415,15 +400,15 @@ nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 	ds->ds_port = port;
 	strncpy(ds->r_addr, r_addr, len);
 	atomic_set(&ds->ds_count, 1);
-	INIT_HLIST_NODE(&ds->ds_node);
+	INIT_LIST_HEAD(&ds->ds_node);
 	ds->ds_clp = NULL;
 
-	write_lock(&hlist->dev_lock);
-	tmp_ds = _data_server_lookup(hlist, ip_addr, port);
+	spin_lock(&nfs4_ds_cache_lock);
+	tmp_ds = _data_server_lookup(ip_addr, port);
 	if (tmp_ds == NULL) {
 		dprintk("%s add new data server ip 0x%x\n", __func__,
 				ds->ds_ip_addr);
-		_data_server_add(hlist, ds);
+		list_add(&ds->ds_node, &nfs4_data_server_cache);
 		*dsp = ds;
 	}
 	if (tmp_ds != NULL) {
@@ -434,7 +419,7 @@ nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 				atomic_read(&tmp_ds->ds_count));
 		*dsp = tmp_ds;
 	}
-	write_unlock(&hlist->dev_lock);
+	spin_unlock(&nfs4_ds_cache_lock);
 }
 
 static struct nfs4_pnfs_ds *
-- 
1.6.6

[PATCH 14/14] SQUASHME pnfs_submit: remove kfree from under spinlock

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4filelayoutdev.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
index 50e2865..5c13dc5 100644
--- a/fs/nfs/nfs4filelayoutdev.c
+++ b/fs/nfs/nfs4filelayoutdev.c
@@ -410,9 +410,7 @@ nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 				ds->ds_ip_addr);
 		list_add(&ds->ds_node, &nfs4_data_server_cache);
 		*dsp = ds;
-	}
-	if (tmp_ds != NULL) {
-		destroy_ds(ds);
+	} else {
 		atomic_inc(&tmp_ds->ds_count);
 		dprintk("%s data server found ip 0x%x, inc'ed ds_count to %d\n",
 				__func__, tmp_ds->ds_ip_addr,
@@ -420,6 +418,8 @@ nfs4_pnfs_ds_add(struct inode *inode, struct nfs4_pnfs_ds **dsp,
 		*dsp = tmp_ds;
 	}
 	spin_unlock(&nfs4_ds_cache_lock);
+	if (tmp_ds != NULL)
+		destroy_ds(ds);
 }
 
 static struct nfs4_pnfs_ds *
-- 
1.6.6

Re: [pnfs] [PATCH 0/14] pnfs_submit file layout client device management

[J. Bruce Fields]

On Thu, Apr 08, 2010 at 03:53:56PM -0400, andros@netapp.com wrote:
> 
> Apply against 2.6.34-rc3 pnfs branch. For pNFS client submission. Only touches
> the file layout driver.
> 
> This set of patches implements several design changes enmerated at
> http://wiki.linux-nfs.org/wiki/index.php/Porposed_Device_ Manangement_Design.

(Should be:

http://wiki.linux-nfs.org/wiki/index.php/Proposed_Device_Management_Design

.)

> 
> - Only call GETDEVICEINFO from one place
> - Only cache the first data server in the multipath_list4 array.
> - Wait to connect to data server when needed for I/O
> - Move data server cache to stand alone cache
> 
> Also some cleanup, and function interfaces are changed in 
> preparation for the soon to come generic deviceid cache.
> 
> 0001-SQUASHME-pnfs_submit-remove-unused-getdevicelist-fun.patch
> 0002-SQUASHME-pnfs-submit-call-get_device_info-from-filel.patch
> 0003-SQUASHME-pnfs_submit-remove-multilist4-caching.patch
> 0004-SQUASHME-pnfs_submit-wait-for-I-O-for-data-server-co.patch
> 0005-SQUASHME-pnfs_submit-remove-xdr-macros.patch
> 0006-SQUASHME-pnfs_submit-remove-unused-nfs4_pnfs_device_.patch
> 0007-SQUASHME-pnfs_submit-change-get_device_info-paramete.patch
> 0008-SQUASHME-pnfs_submit-change-decode_and_add_device-pa.patch
> 0009-SQUASHME-pnfs_submit-change-decode_device-parameter.patch
> 0010-SQUASHME-pnfs_submit-change-decode_and_add_ds-parame.patch
> 0011-SQUASHME-pnfs_submit-change-nfs4_pnfs_ds_add-paramet.patch
> 0012-SQUASHME-pnfs_submit-remove-filelayout-CB_NOTIFY_DEV.patch
> 0013-SQUASHME-pnfs_submit-stand-alone-data-server-cache.patch
> 0014-SQUASHME-pnfs_submit-remove-kfree-from-under-spinloc.patch
> 
> Testing:
> CONFIG_NFS_V4_1 set:
> pNFS mount, NFSv4.1 mount, NFSv4.0 mount:
> Connecatathon tests pass. pyNFS testclient tests pass.
> 
> CONFIG_NFS_V4_1 not set:
> Connecatahon tests pass.
> 
> 
> -->Andy
> 
> _______________________________________________
> pNFS mailing list
> pNFS@linux-nfs.org
> http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

Re: [pnfs] [PATCH 02/14] SQUASHME pnfs-submit: call get_device_info from filelayout_check

[Benny Halevy]

On Apr. 08, 2010, 22:53 +0300, andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> filelayout_check() validates a LAYOUTGET return including the deviceid.
> If the deviceid is not cached, GETDEVICEINFO is called.
> 
> Once cached, the deviceid will only be uncached upon umount or a return of all
> refering layouts. So, there is no need for a get_device_info call in the I/O
> paths.
> 
> Remove nfs4_file_layout_dsaddr_get.
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/nfs4filelayout.c    |   11 +++++++----
>  fs/nfs/nfs4filelayout.h    |    6 ++++--
>  fs/nfs/nfs4filelayoutdev.c |   27 +++------------------------
>  3 files changed, 14 insertions(+), 30 deletions(-)
> 
> diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
> index df6c1d0..ffc3ef2 100644
> --- a/fs/nfs/nfs4filelayout.c
> +++ b/fs/nfs/nfs4filelayout.c
> @@ -413,12 +413,15 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
>  	struct nfs_server *nfss = NFS_SERVER(PNFS_INODE(lo));
>  
>  	dprintk("--> %s\n", __func__);
> -	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(PNFS_INODE(lo)),
> +	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
>  					     &fl->dev_id);
>  	if (dsaddr == NULL) {
> -		dprintk("%s NO device for dev_id %s\n",
> +		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
> +		if (dsaddr == NULL) {
> +			dprintk("%s NO device for dev_id %s\n",
>  				__func__, deviceid_fmt(&fl->dev_id));
> -		goto out;
> +			goto out;
> +		}
>  	}
>  	if (fl->first_stripe_index < 0 ||
>  	    fl->first_stripe_index > dsaddr->stripe_count) {
> @@ -645,7 +648,7 @@ filelayout_commit(struct pnfs_layout_type *layoutid, int sync,
>  	stripesz = filelayout_get_stripesize(layoutid);
>  	dprintk("%s stripesize %Zd\n", __func__, stripesz);
>  
> -	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(data->inode),
> +	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(data->inode)->hlist,
>  					     &nfslay->dev_id);
>  	if (dsaddr == NULL) {
>  		data->pdata.pnfs_error = -EIO;
> diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
> index d003809..3821989 100644
> --- a/fs/nfs/nfs4filelayout.h
> +++ b/fs/nfs/nfs4filelayout.h
> @@ -121,7 +121,9 @@ u32 filelayout_dserver_get_index(loff_t offset,
>  } while (0)
>  
>  struct nfs4_file_layout_dsaddr *
> -nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *,
> -			  struct pnfs_deviceid *);
> +nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
> +			   struct pnfs_deviceid *dev_id);
> +struct nfs4_file_layout_dsaddr *
> +get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
>  

Andy, this function name is too generic IMO.
better be something more specific to pnfs and file layout.
(I'm committing this patchset as is so we can fix that later)

Benny

>  #endif /* FS_NFS_NFS4FILELAYOUT_H */
> diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
> index e557679..ebb9845 100644
> --- a/fs/nfs/nfs4filelayoutdev.c
> +++ b/fs/nfs/nfs4filelayoutdev.c
> @@ -52,13 +52,6 @@
>  
>  #define NFSDBG_FACILITY		NFSDBG_PNFS_LD
>  
> -struct nfs4_file_layout_dsaddr *nfs4_file_layout_dsaddr_get(
> -					struct filelayout_mount_type *mt,
> -					struct pnfs_deviceid *dev_id);
> -struct nfs4_file_layout_dsaddr *nfs4_pnfs_device_item_find(
> -					struct nfs4_pnfs_dev_hlist *hlist,
> -					struct pnfs_deviceid *dev_id);
> -
>  void
>  print_ds_list(struct nfs4_multipath *multipath)
>  {
> @@ -665,7 +658,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
>  /* Retrieve the information for dev_id, add it to the list
>   * of available devices, and return it.
>   */
> -static struct nfs4_file_layout_dsaddr *
> +struct nfs4_file_layout_dsaddr *
>  get_device_info(struct filelayout_mount_type *mt,
>  		struct pnfs_deviceid *dev_id)
>  {
> @@ -737,21 +730,6 @@ out_free:
>  }
>  
>  struct nfs4_file_layout_dsaddr *
> -nfs4_file_layout_dsaddr_get(struct filelayout_mount_type *mt,
> -			    struct pnfs_deviceid *dev_id)
> -{
> -	struct nfs4_file_layout_dsaddr *dsaddr;
> -
> -	read_lock(&mt->hlist->dev_lock);
> -	dsaddr = _device_lookup(mt->hlist, dev_id);
> -	read_unlock(&mt->hlist->dev_lock);
> -
> -	if (dsaddr == NULL)
> -		dsaddr = get_device_info(mt, dev_id);
> -	return dsaddr;
> -}
> -
> -struct nfs4_file_layout_dsaddr *
>  nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
>  			   struct pnfs_deviceid *dev_id)
>  {
> @@ -798,7 +776,8 @@ nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
>  	if (!layout)
>  		return 1;
>  
> -	dsaddr = nfs4_file_layout_dsaddr_get(FILE_MT(inode), &layout->dev_id);
> +	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(inode)->hlist,
> +					    &layout->dev_id);
>  	if (dsaddr == NULL)
>  		return 1;
>  

Re: [pnfs] [PATCH 07/14] SQUASHME pnfs_submit: change get_device_info parameter

[Benny Halevy]

On Apr. 08, 2010, 22:54 +0300, andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> In preparation to use the generic deviceid cache.
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/nfs4filelayout.c    |    2 +-
>  fs/nfs/nfs4filelayout.h    |    2 +-
>  fs/nfs/nfs4filelayoutdev.c |    9 ++++-----
>  3 files changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
> index 20b3ae2..f222c3b 100644
> --- a/fs/nfs/nfs4filelayout.c
> +++ b/fs/nfs/nfs4filelayout.c
> @@ -399,7 +399,7 @@ filelayout_check_layout(struct pnfs_layout_type *lo,
>  	dsaddr = nfs4_pnfs_device_item_find(FILE_MT(PNFS_INODE(lo))->hlist,
>  					     &fl->dev_id);
>  	if (dsaddr == NULL) {
> -		dsaddr = get_device_info(FILE_MT(PNFS_INODE(lo)), &fl->dev_id);
> +		dsaddr = get_device_info(PNFS_INODE(lo), &fl->dev_id);
>  		if (dsaddr == NULL) {
>  			dprintk("%s NO device for dev_id %s\n",
>  				__func__, deviceid_fmt(&fl->dev_id));
> diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
> index 04cca7e..560f7f6 100644
> --- a/fs/nfs/nfs4filelayout.h
> +++ b/fs/nfs/nfs4filelayout.h
> @@ -109,6 +109,6 @@ struct nfs4_file_layout_dsaddr *
>  nfs4_pnfs_device_item_find(struct nfs4_pnfs_dev_hlist *hlist,
>  			   struct pnfs_deviceid *dev_id);
>  struct nfs4_file_layout_dsaddr *
> -get_device_info(struct filelayout_mount_type *mt, struct pnfs_deviceid *dev_id);
> +get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id);
>  
>  #endif /* FS_NFS_NFS4FILELAYOUT_H */
> diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
> index db6817d..9844a9c 100644
> --- a/fs/nfs/nfs4filelayoutdev.c
> +++ b/fs/nfs/nfs4filelayoutdev.c
> @@ -623,8 +623,7 @@ decode_and_add_device(struct filelayout_mount_type *mt, struct pnfs_device *dev)
>   * of available devices, and return it.
>   */
>  struct nfs4_file_layout_dsaddr *
> -get_device_info(struct filelayout_mount_type *mt,
> -		struct pnfs_deviceid *dev_id)
> +get_device_info(struct inode *inode, struct pnfs_deviceid *dev_id)
>  {
>  	struct pnfs_device *pdev = NULL;
>  	int maxpages = NFS4_GETDEVINFO_MAXSIZE >> PAGE_SHIFT;
> @@ -632,7 +631,7 @@ get_device_info(struct filelayout_mount_type *mt,
>  	struct nfs4_file_layout_dsaddr *dsaddr = NULL;
>  	int rc, i, j, minpages = 1;
>  
> -	dprintk("%s mt %p\n", __func__, mt);
> +	dprintk("%s \n", __func__);

nit: checkpatch rightfully complains about the stray space character before the newline.
I'll just remove that upon committing, but there are other places
like that we need to clean up before submitting the final version...

Benny

>  	pdev = kzalloc(sizeof(struct pnfs_device), GFP_KERNEL);
>  	if (pdev == NULL)
>  		return NULL;
> @@ -664,7 +663,7 @@ retry_once:
>  	/* TODO: Update types when CB_NOTIFY_DEVICEID is available */
>  	pdev->dev_notify_types = 0;
>  
> -	rc = pnfs_callback_ops->nfs_getdeviceinfo(mt->fl_sb, pdev);
> +	rc = pnfs_callback_ops->nfs_getdeviceinfo(inode->i_sb, pdev);
>  	/* Retry once with the returned mincount if a page was too small */
>  	dprintk("%s getdevice info returns %d minpages %d\n", __func__, rc,
>  		minpages);
> @@ -682,7 +681,7 @@ retry_once:
>  	/* Found new device, need to decode it and then add it to the
>  	 * list of known devices for this mountpoint.
>  	 */
> -	dsaddr = decode_and_add_device(mt, pdev);
> +	dsaddr = decode_and_add_device(FILE_MT(inode), pdev);
>  out_free:
>  	if (minpages > 1 && pdev->area != NULL)
>  		vunmap(pdev->area);

Re: [pnfs] [PATCH 12/14] SQUASHME pnfs_submit: remove filelayout CB_NOTIFY_DEVICE support

[Benny Halevy]

OK. I'll keep a patch reverting this patch (reintroducing this functionality)
in the pnfs branch, on top of the pnfs-submit branch.

Benny

andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> We won't support CB_NOTIFY_DEVICE in the first submission stage.
> Remove filelayout_device_delete and associated functionality.
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/nfs4filelayout.c    |   16 ----------------
>  fs/nfs/nfs4filelayout.h    |    2 --
>  fs/nfs/nfs4filelayoutdev.c |   18 ------------------
>  3 files changed, 0 insertions(+), 36 deletions(-)
> 
> diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
> index f222c3b..0530b59 100644
> --- a/fs/nfs/nfs4filelayout.c
> +++ b/fs/nfs/nfs4filelayout.c
> @@ -135,21 +135,6 @@ filelayout_uninitialize_mountpoint(struct pnfs_mount_type *mountid)
>  	return 0;
>  }
>  
> -int
> -filelayout_device_delete(struct pnfs_mount_type *mountid,
> -				struct pnfs_deviceid *dev_id)
> -{
> -	struct filelayout_mount_type *fl_mt = NULL;
> -
> -	if (mountid) {
> -		fl_mt = (struct filelayout_mount_type *)mountid->mountid;
> -
> -		if (fl_mt != NULL)
> -			nfs4_pnfs_dev_destroy(fl_mt->hlist, dev_id);
> -	}
> -	return 0;
> -}
> -
>  /* This function is used by the layout driver to calculate the
>   * offset of the file on the dserver based on whether the
>   * layout type is STRIPE_DENSE or STRIPE_SPARSE
> @@ -797,7 +782,6 @@ struct layoutdriver_io_operations filelayout_io_operations = {
>  	.free_lseg               = filelayout_free_lseg,
>  	.initialize_mountpoint   = filelayout_initialize_mountpoint,
>  	.uninitialize_mountpoint = filelayout_uninitialize_mountpoint,
> -	.device_delete           = filelayout_device_delete,
>  };
>  
>  struct layoutdriver_policy_operations filelayout_policy_operations = {
> diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/nfs4filelayout.h
> index 560f7f6..e988891 100644
> --- a/fs/nfs/nfs4filelayout.h
> +++ b/fs/nfs/nfs4filelayout.h
> @@ -96,8 +96,6 @@ extern void print_ds(struct nfs4_pnfs_ds *ds);
>  char *deviceid_fmt(const struct pnfs_deviceid *dev_id);
>  int  nfs4_pnfs_devlist_init(struct nfs4_pnfs_dev_hlist *hlist);
>  void nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist);
> -void nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
> -				struct pnfs_deviceid *dev_id);
>  int nfs4_pnfs_dserver_get(struct pnfs_layout_segment *lseg,
>  			  loff_t offset,
>  			  size_t count,
> diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/nfs4filelayoutdev.c
> index 2cbe931..2d98ca3 100644
> --- a/fs/nfs/nfs4filelayoutdev.c
> +++ b/fs/nfs/nfs4filelayoutdev.c
> @@ -368,24 +368,6 @@ nfs4_pnfs_devlist_destroy(struct nfs4_pnfs_dev_hlist *hlist)
>  	}
>  }
>  
> -/* De-alloc a devices for a mount point. */
> -void
> -nfs4_pnfs_dev_destroy(struct nfs4_pnfs_dev_hlist *hlist,
> -			struct pnfs_deviceid *dev_id)
> -{
> -	struct nfs4_file_layout_dsaddr *dsaddr;
> -
> -	if (hlist == NULL)
> -		return;
> -
> -	dprintk("%s: dev_id=%s\n", __func__, deviceid_fmt(dev_id));
> -
> -	dsaddr = nfs4_pnfs_device_item_find(hlist, dev_id);
> -	if (dsaddr)
> -		/* nfs4_pnfs_device_destroy grabs hlist->dev_lock */
> -		nfs4_pnfs_device_destroy(dsaddr, hlist);
> -}
> -
>  /*
>   * Add the device to the list of available devices for this mount point.
>   * The * rpc client is created during first I/O.

Re: [pnfs] [PATCH 0/14] pnfs_submit file layout client device management

[Benny Halevy]

Committed to a new branch in the 2.6.34-rc3 stream: pnfs-submit

Thanks!

Benny

On Apr. 08, 2010, 22:53 +0300, andros@netapp.com wrote:
> 
> Apply against 2.6.34-rc3 pnfs branch. For pNFS client submission. Only touches
> the file layout driver.
> 
> This set of patches implements several design changes enmerated at
> http://wiki.linux-nfs.org/wiki/index.php/Porposed_Device_ Manangement_Design.
> 
> - Only call GETDEVICEINFO from one place
> - Only cache the first data server in the multipath_list4 array.
> - Wait to connect to data server when needed for I/O
> - Move data server cache to stand alone cache
> 
> Also some cleanup, and function interfaces are changed in 
> preparation for the soon to come generic deviceid cache.
> 
> 0001-SQUASHME-pnfs_submit-remove-unused-getdevicelist-fun.patch
> 0002-SQUASHME-pnfs-submit-call-get_device_info-from-filel.patch
> 0003-SQUASHME-pnfs_submit-remove-multilist4-caching.patch
> 0004-SQUASHME-pnfs_submit-wait-for-I-O-for-data-server-co.patch
> 0005-SQUASHME-pnfs_submit-remove-xdr-macros.patch
> 0006-SQUASHME-pnfs_submit-remove-unused-nfs4_pnfs_device_.patch
> 0007-SQUASHME-pnfs_submit-change-get_device_info-paramete.patch
> 0008-SQUASHME-pnfs_submit-change-decode_and_add_device-pa.patch
> 0009-SQUASHME-pnfs_submit-change-decode_device-parameter.patch
> 0010-SQUASHME-pnfs_submit-change-decode_and_add_ds-parame.patch
> 0011-SQUASHME-pnfs_submit-change-nfs4_pnfs_ds_add-paramet.patch
> 0012-SQUASHME-pnfs_submit-remove-filelayout-CB_NOTIFY_DEV.patch
> 0013-SQUASHME-pnfs_submit-stand-alone-data-server-cache.patch
> 0014-SQUASHME-pnfs_submit-remove-kfree-from-under-spinloc.patch
> 
> Testing:
> CONFIG_NFS_V4_1 set:
> pNFS mount, NFSv4.1 mount, NFSv4.0 mount:
> Connecatathon tests pass. pyNFS testclient tests pass.
> 
> CONFIG_NFS_V4_1 not set:
> Connecatahon tests pass.
> 
> 
> -->Andy
> 
> _______________________________________________
> pNFS mailing list
> pNFS@linux-nfs.org
> http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs