[PATCH 0/7] pnfs wave 4 proposal

[PATCH 0/7] pnfs wave 4 proposal

[Fred Isaman]

These apply on top of Andy's wave3-on-nfs-for-next-rev4.

These patches give write functionality to the pnfs file layout driver.
The primary limitation is that we have not yet modified the commit
code, so we can not yet handle COMMITs to the data server.  We get
around this for the moment by sending any WRITE to a data server which
does not have commit-to-mds set as a SYNCH write.

Fred

[PATCH 1/7] wave4: nfs: rearrange nfs_write_rpcsetup

[Fred Isaman]

From: The pNFS Team <linux-nfs@vger.kernel.org>

Reorder nfs_write_rpcsetup, preparing for a pnfs switch.

FIXME: these signoffs are ridiculous for this simple patch.
Lets just pick an author and be done with it.

Reported-by: Alexandros Batsakis <batsakis@netapp.com>
Signed-off-by: Andy Adamson <andros@citi.umich.edu>
Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
Signed-off-by: Fred Isaman <iisaman@citi.umich.edu>
Signed-off-by: J. Bruce Fields <bfields@fieldses.org>
Signed-off-by: Mike Sager <sager@netapp.com>
Signed-off-by: Ricardo Labiaga <Ricardo.Labiaga@netapp.com>
Signed-off-by: Tao Guo <guotao@nrchpc.ac.cn>
Signed-off-by: Andy Adamson <andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
Signed-off-by: Fred Isaman <iisaman@netapp.com>
---
 fs/nfs/write.c |   82 +++++++++++++++++++++++++++++++------------------------
 1 files changed, 46 insertions(+), 36 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index aca0268..5604854 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -782,25 +782,21 @@ static int flush_task_priority(int how)
 	return RPC_PRIORITY_NORMAL;
 }
 
-/*
- * Set up the argument/result storage required for the RPC call.
- */
-static int nfs_write_rpcsetup(struct nfs_page *req,
-		struct nfs_write_data *data,
-		const struct rpc_call_ops *call_ops,
-		unsigned int count, unsigned int offset,
-		int how)
+static int nfs_initiate_write(struct nfs_write_data *data,
+		       struct rpc_clnt *clnt,
+		       const struct rpc_call_ops *call_ops,
+		       int how)
 {
-	struct inode *inode = req->wb_context->path.dentry->d_inode;
+	struct inode *inode = data->inode;
 	int priority = flush_task_priority(how);
 	struct rpc_task *task;
 	struct rpc_message msg = {
 		.rpc_argp = &data->args,
 		.rpc_resp = &data->res,
-		.rpc_cred = req->wb_context->cred,
+		.rpc_cred = data->cred,
 	};
 	struct rpc_task_setup task_setup_data = {
-		.rpc_client = NFS_CLIENT(inode),
+		.rpc_client = clnt,
 		.task = &data->task,
 		.rpc_message = &msg,
 		.callback_ops = call_ops,
@@ -811,12 +807,49 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
 	};
 	int ret = 0;
 
+	/* Set up the initial task struct.  */
+	NFS_PROTO(inode)->write_setup(data, &msg);
+
+	dprintk("NFS: %5u initiated write call "
+		"(req %s/%lld, %u bytes @ offset %llu)\n",
+		data->task.tk_pid,
+		inode->i_sb->s_id,
+		(long long)NFS_FILEID(inode),
+		data->args.count,
+		(unsigned long long)data->args.offset);
+
+	task = rpc_run_task(&task_setup_data);
+	if (IS_ERR(task)) {
+		ret = PTR_ERR(task);
+		goto out;
+	}
+	if (how & FLUSH_SYNC) {
+		ret = rpc_wait_for_completion_task(task);
+		if (ret == 0)
+			ret = task->tk_status;
+	}
+	rpc_put_task(task);
+out:
+	return ret;
+}
+
+/*
+ * Set up the argument/result storage required for the RPC call.
+ */
+static int nfs_write_rpcsetup(struct nfs_page *req,
+		struct nfs_write_data *data,
+		const struct rpc_call_ops *call_ops,
+		unsigned int count, unsigned int offset,
+		int how)
+{
+	struct inode *inode = req->wb_context->path.dentry->d_inode;
+
 	/* Set up the RPC argument and reply structs
 	 * NB: take care not to mess about with data->commit et al. */
 
 	data->req = req;
 	data->inode = inode = req->wb_context->path.dentry->d_inode;
-	data->cred = msg.rpc_cred;
+	data->cred = req->wb_context->cred;
 
 	data->args.fh     = NFS_FH(inode);
 	data->args.offset = req_offset(req) + offset;
@@ -837,30 +870,7 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
 	data->res.verf    = &data->verf;
 	nfs_fattr_init(&data->fattr);
 
-	/* Set up the initial task struct.  */
-	NFS_PROTO(inode)->write_setup(data, &msg);
-
-	dprintk("NFS: %5u initiated write call "
-		"(req %s/%lld, %u bytes @ offset %llu)\n",
-		data->task.tk_pid,
-		inode->i_sb->s_id,
-		(long long)NFS_FILEID(inode),
-		count,
-		(unsigned long long)data->args.offset);
-
-	task = rpc_run_task(&task_setup_data);
-	if (IS_ERR(task)) {
-		ret = PTR_ERR(task);
-		goto out;
-	}
-	if (how & FLUSH_SYNC) {
-		ret = rpc_wait_for_completion_task(task);
-		if (ret == 0)
-			ret = task->tk_status;
-	}
-	rpc_put_task(task);
-out:
-	return ret;
+	return nfs_initiate_write(data, NFS_CLIENT(inode), call_ops, how);
 }
 
 /* If a nfs_flush_* function fails, it should remove reqs from @head and
-- 
1.7.2.1

Re: [PATCH 1/7] wave4: nfs: rearrange nfs_write_rpcsetup

[Andy Adamson]

Since I'm the author for the read version of this code, I'll raise my hand to be the author for this version.

-->Andy

On Feb 11, 2011, at 11:54 AM, Fred Isaman wrote:

> From: The pNFS Team <linux-nfs@vger.kernel.org>
> 
> Reorder nfs_write_rpcsetup, preparing for a pnfs switch.
> 
> FIXME: these signoffs are ridiculous for this simple patch.
> Lets just pick an author and be done with it.
> 
> Reported-by: Alexandros Batsakis <batsakis@netapp.com>
> Signed-off-by: Andy Adamson <andros@citi.umich.edu>
> Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
> Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
> Signed-off-by: Fred Isaman <iisaman@citi.umich.edu>
> Signed-off-by: J. Bruce Fields <bfields@fieldses.org>
> Signed-off-by: Mike Sager <sager@netapp.com>
> Signed-off-by: Ricardo Labiaga <Ricardo.Labiaga@netapp.com>
> Signed-off-by: Tao Guo <guotao@nrchpc.ac.cn>
> Signed-off-by: Andy Adamson <andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> Signed-off-by: Fred Isaman <iisaman@netapp.com>
> ---
> fs/nfs/write.c |   82 +++++++++++++++++++++++++++++++------------------------
> 1 files changed, 46 insertions(+), 36 deletions(-)
> 
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index aca0268..5604854 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -782,25 +782,21 @@ static int flush_task_priority(int how)
> 	return RPC_PRIORITY_NORMAL;
> }
> 
> -/*
> - * Set up the argument/result storage required for the RPC call.
> - */
> -static int nfs_write_rpcsetup(struct nfs_page *req,
> -		struct nfs_write_data *data,
> -		const struct rpc_call_ops *call_ops,
> -		unsigned int count, unsigned int offset,
> -		int how)
> +static int nfs_initiate_write(struct nfs_write_data *data,
> +		       struct rpc_clnt *clnt,
> +		       const struct rpc_call_ops *call_ops,
> +		       int how)
> {
> -	struct inode *inode = req->wb_context->path.dentry->d_inode;
> +	struct inode *inode = data->inode;
> 	int priority = flush_task_priority(how);
> 	struct rpc_task *task;
> 	struct rpc_message msg = {
> 		.rpc_argp = &data->args,
> 		.rpc_resp = &data->res,
> -		.rpc_cred = req->wb_context->cred,
> +		.rpc_cred = data->cred,
> 	};
> 	struct rpc_task_setup task_setup_data = {
> -		.rpc_client = NFS_CLIENT(inode),
> +		.rpc_client = clnt,
> 		.task = &data->task,
> 		.rpc_message = &msg,
> 		.callback_ops = call_ops,
> @@ -811,12 +807,49 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
> 	};
> 	int ret = 0;
> 
> +	/* Set up the initial task struct.  */
> +	NFS_PROTO(inode)->write_setup(data, &msg);
> +
> +	dprintk("NFS: %5u initiated write call "
> +		"(req %s/%lld, %u bytes @ offset %llu)\n",
> +		data->task.tk_pid,
> +		inode->i_sb->s_id,
> +		(long long)NFS_FILEID(inode),
> +		data->args.count,
> +		(unsigned long long)data->args.offset);
> +
> +	task = rpc_run_task(&task_setup_data);
> +	if (IS_ERR(task)) {
> +		ret = PTR_ERR(task);
> +		goto out;
> +	}
> +	if (how & FLUSH_SYNC) {
> +		ret = rpc_wait_for_completion_task(task);
> +		if (ret == 0)
> +			ret = task->tk_status;
> +	}
> +	rpc_put_task(task);
> +out:
> +	return ret;
> +}
> +
> +/*
> + * Set up the argument/result storage required for the RPC call.
> + */
> +static int nfs_write_rpcsetup(struct nfs_page *req,
> +		struct nfs_write_data *data,
> +		const struct rpc_call_ops *call_ops,
> +		unsigned int count, unsigned int offset,
> +		int how)
> +{
> +	struct inode *inode = req->wb_context->path.dentry->d_inode;
> +
> 	/* Set up the RPC argument and reply structs
> 	 * NB: take care not to mess about with data->commit et al. */
> 
> 	data->req = req;
> 	data->inode = inode = req->wb_context->path.dentry->d_inode;
> -	data->cred = msg.rpc_cred;
> +	data->cred = req->wb_context->cred;
> 
> 	data->args.fh     = NFS_FH(inode);
> 	data->args.offset = req_offset(req) + offset;
> @@ -837,30 +870,7 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
> 	data->res.verf    = &data->verf;
> 	nfs_fattr_init(&data->fattr);
> 
> -	/* Set up the initial task struct.  */
> -	NFS_PROTO(inode)->write_setup(data, &msg);
> -
> -	dprintk("NFS: %5u initiated write call "
> -		"(req %s/%lld, %u bytes @ offset %llu)\n",
> -		data->task.tk_pid,
> -		inode->i_sb->s_id,
> -		(long long)NFS_FILEID(inode),
> -		count,
> -		(unsigned long long)data->args.offset);
> -
> -	task = rpc_run_task(&task_setup_data);
> -	if (IS_ERR(task)) {
> -		ret = PTR_ERR(task);
> -		goto out;
> -	}
> -	if (how & FLUSH_SYNC) {
> -		ret = rpc_wait_for_completion_task(task);
> -		if (ret == 0)
> -			ret = task->tk_status;
> -	}
> -	rpc_put_task(task);
> -out:
> -	return ret;
> +	return nfs_initiate_write(data, NFS_CLIENT(inode), call_ops, how);
> }
> 
> /* If a nfs_flush_* function fails, it should remove reqs from @head and
> -- 
> 1.7.2.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] wave4: nfs: rearrange nfs_write_rpcsetup

[Fred Isaman]

On Feb 14, 2011, at 12:22 PM, Andy Adamson wrote:

> Since I'm the author for the read version of this code, I'll raise my hand to be the author for this version.
> 
> -->Andy
> 

OK, do you wan't patch 5 as well?...they were both split from the same "pNFS Team" patch.

Fred

> On Feb 11, 2011, at 11:54 AM, Fred Isaman wrote:
> 
>> From: The pNFS Team <linux-nfs@vger.kernel.org>
>> 
>> Reorder nfs_write_rpcsetup, preparing for a pnfs switch.
>> 
>> FIXME: these signoffs are ridiculous for this simple patch.
>> Lets just pick an author and be done with it.
>> 
>> Reported-by: Alexandros Batsakis <batsakis@netapp.com>
>> Signed-off-by: Andy Adamson <andros@citi.umich.edu>
>> Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
>> Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
>> Signed-off-by: Fred Isaman <iisaman@citi.umich.edu>
>> Signed-off-by: J. Bruce Fields <bfields@fieldses.org>
>> Signed-off-by: Mike Sager <sager@netapp.com>
>> Signed-off-by: Ricardo Labiaga <Ricardo.Labiaga@netapp.com>
>> Signed-off-by: Tao Guo <guotao@nrchpc.ac.cn>
>> Signed-off-by: Andy Adamson <andros@netapp.com>
>> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
>> Signed-off-by: Fred Isaman <iisaman@netapp.com>
>> ---
>> fs/nfs/write.c |   82 +++++++++++++++++++++++++++++++------------------------
>> 1 files changed, 46 insertions(+), 36 deletions(-)
>> 
>> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
>> index aca0268..5604854 100644
>> --- a/fs/nfs/write.c
>> +++ b/fs/nfs/write.c
>> @@ -782,25 +782,21 @@ static int flush_task_priority(int how)
>> 	return RPC_PRIORITY_NORMAL;
>> }
>> 
>> -/*
>> - * Set up the argument/result storage required for the RPC call.
>> - */
>> -static int nfs_write_rpcsetup(struct nfs_page *req,
>> -		struct nfs_write_data *data,
>> -		const struct rpc_call_ops *call_ops,
>> -		unsigned int count, unsigned int offset,
>> -		int how)
>> +static int nfs_initiate_write(struct nfs_write_data *data,
>> +		       struct rpc_clnt *clnt,
>> +		       const struct rpc_call_ops *call_ops,
>> +		       int how)
>> {
>> -	struct inode *inode = req->wb_context->path.dentry->d_inode;
>> +	struct inode *inode = data->inode;
>> 	int priority = flush_task_priority(how);
>> 	struct rpc_task *task;
>> 	struct rpc_message msg = {
>> 		.rpc_argp = &data->args,
>> 		.rpc_resp = &data->res,
>> -		.rpc_cred = req->wb_context->cred,
>> +		.rpc_cred = data->cred,
>> 	};
>> 	struct rpc_task_setup task_setup_data = {
>> -		.rpc_client = NFS_CLIENT(inode),
>> +		.rpc_client = clnt,
>> 		.task = &data->task,
>> 		.rpc_message = &msg,
>> 		.callback_ops = call_ops,
>> @@ -811,12 +807,49 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
>> 	};
>> 	int ret = 0;
>> 
>> +	/* Set up the initial task struct.  */
>> +	NFS_PROTO(inode)->write_setup(data, &msg);
>> +
>> +	dprintk("NFS: %5u initiated write call "
>> +		"(req %s/%lld, %u bytes @ offset %llu)\n",
>> +		data->task.tk_pid,
>> +		inode->i_sb->s_id,
>> +		(long long)NFS_FILEID(inode),
>> +		data->args.count,
>> +		(unsigned long long)data->args.offset);
>> +
>> +	task = rpc_run_task(&task_setup_data);
>> +	if (IS_ERR(task)) {
>> +		ret = PTR_ERR(task);
>> +		goto out;
>> +	}
>> +	if (how & FLUSH_SYNC) {
>> +		ret = rpc_wait_for_completion_task(task);
>> +		if (ret == 0)
>> +			ret = task->tk_status;
>> +	}
>> +	rpc_put_task(task);
>> +out:
>> +	return ret;
>> +}
>> +
>> +/*
>> + * Set up the argument/result storage required for the RPC call.
>> + */
>> +static int nfs_write_rpcsetup(struct nfs_page *req,
>> +		struct nfs_write_data *data,
>> +		const struct rpc_call_ops *call_ops,
>> +		unsigned int count, unsigned int offset,
>> +		int how)
>> +{
>> +	struct inode *inode = req->wb_context->path.dentry->d_inode;
>> +
>> 	/* Set up the RPC argument and reply structs
>> 	 * NB: take care not to mess about with data->commit et al. */
>> 
>> 	data->req = req;
>> 	data->inode = inode = req->wb_context->path.dentry->d_inode;
>> -	data->cred = msg.rpc_cred;
>> +	data->cred = req->wb_context->cred;
>> 
>> 	data->args.fh     = NFS_FH(inode);
>> 	data->args.offset = req_offset(req) + offset;
>> @@ -837,30 +870,7 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
>> 	data->res.verf    = &data->verf;
>> 	nfs_fattr_init(&data->fattr);
>> 
>> -	/* Set up the initial task struct.  */
>> -	NFS_PROTO(inode)->write_setup(data, &msg);
>> -
>> -	dprintk("NFS: %5u initiated write call "
>> -		"(req %s/%lld, %u bytes @ offset %llu)\n",
>> -		data->task.tk_pid,
>> -		inode->i_sb->s_id,
>> -		(long long)NFS_FILEID(inode),
>> -		count,
>> -		(unsigned long long)data->args.offset);
>> -
>> -	task = rpc_run_task(&task_setup_data);
>> -	if (IS_ERR(task)) {
>> -		ret = PTR_ERR(task);
>> -		goto out;
>> -	}
>> -	if (how & FLUSH_SYNC) {
>> -		ret = rpc_wait_for_completion_task(task);
>> -		if (ret == 0)
>> -			ret = task->tk_status;
>> -	}
>> -	rpc_put_task(task);
>> -out:
>> -	return ret;
>> +	return nfs_initiate_write(data, NFS_CLIENT(inode), call_ops, how);
>> }
>> 
>> /* If a nfs_flush_* function fails, it should remove reqs from @head and
>> -- 
>> 1.7.2.1
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH 1/7] wave4: nfs: rearrange nfs_write_rpcsetup

[William A. (Andy) Adamson]

sure.

On Mon, Feb 14, 2011 at 12:27 PM, Fred Isaman <iisaman@netapp.com> wrote:
>
> On Feb 14, 2011, at 12:22 PM, Andy Adamson wrote:
>
>> Since I'm the author for the read version of this code, I'll raise my hand to be the author for this version.
>>
>> -->Andy
>>
>
> OK, do you wan't patch 5 as well?...they were both split from the same "pNFS Team" patch.
>
> Fred
>
>> On Feb 11, 2011, at 11:54 AM, Fred Isaman wrote:
>>
>>> From: The pNFS Team <linux-nfs@vger.kernel.org>
>>>
>>> Reorder nfs_write_rpcsetup, preparing for a pnfs switch.
>>>
>>> FIXME: these signoffs are ridiculous for this simple patch.
>>> Lets just pick an author and be done with it.
>>>
>>> Reported-by: Alexandros Batsakis <batsakis@netapp.com>
>>> Signed-off-by: Andy Adamson <andros@citi.umich.edu>
>>> Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
>>> Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
>>> Signed-off-by: Fred Isaman <iisaman@citi.umich.edu>
>>> Signed-off-by: J. Bruce Fields <bfields@fieldses.org>
>>> Signed-off-by: Mike Sager <sager@netapp.com>
>>> Signed-off-by: Ricardo Labiaga <Ricardo.Labiaga@netapp.com>
>>> Signed-off-by: Tao Guo <guotao@nrchpc.ac.cn>
>>> Signed-off-by: Andy Adamson <andros@netapp.com>
>>> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
>>> Signed-off-by: Fred Isaman <iisaman@netapp.com>
>>> ---
>>> fs/nfs/write.c |   82 +++++++++++++++++++++++++++++++------------------------
>>> 1 files changed, 46 insertions(+), 36 deletions(-)
>>>
>>> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
>>> index aca0268..5604854 100644
>>> --- a/fs/nfs/write.c
>>> +++ b/fs/nfs/write.c
>>> @@ -782,25 +782,21 @@ static int flush_task_priority(int how)
>>>      return RPC_PRIORITY_NORMAL;
>>> }
>>>
>>> -/*
>>> - * Set up the argument/result storage required for the RPC call.
>>> - */
>>> -static int nfs_write_rpcsetup(struct nfs_page *req,
>>> -            struct nfs_write_data *data,
>>> -            const struct rpc_call_ops *call_ops,
>>> -            unsigned int count, unsigned int offset,
>>> -            int how)
>>> +static int nfs_initiate_write(struct nfs_write_data *data,
>>> +                   struct rpc_clnt *clnt,
>>> +                   const struct rpc_call_ops *call_ops,
>>> +                   int how)
>>> {
>>> -    struct inode *inode = req->wb_context->path.dentry->d_inode;
>>> +    struct inode *inode = data->inode;
>>>      int priority = flush_task_priority(how);
>>>      struct rpc_task *task;
>>>      struct rpc_message msg = {
>>>              .rpc_argp = &data->args,
>>>              .rpc_resp = &data->res,
>>> -            .rpc_cred = req->wb_context->cred,
>>> +            .rpc_cred = data->cred,
>>>      };
>>>      struct rpc_task_setup task_setup_data = {
>>> -            .rpc_client = NFS_CLIENT(inode),
>>> +            .rpc_client = clnt,
>>>              .task = &data->task,
>>>              .rpc_message = &msg,
>>>              .callback_ops = call_ops,
>>> @@ -811,12 +807,49 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
>>>      };
>>>      int ret = 0;
>>>
>>> +    /* Set up the initial task struct.  */
>>> +    NFS_PROTO(inode)->write_setup(data, &msg);
>>> +
>>> +    dprintk("NFS: %5u initiated write call "
>>> +            "(req %s/%lld, %u bytes @ offset %llu)\n",
>>> +            data->task.tk_pid,
>>> +            inode->i_sb->s_id,
>>> +            (long long)NFS_FILEID(inode),
>>> +            data->args.count,
>>> +            (unsigned long long)data->args.offset);
>>> +
>>> +    task = rpc_run_task(&task_setup_data);
>>> +    if (IS_ERR(task)) {
>>> +            ret = PTR_ERR(task);
>>> +            goto out;
>>> +    }
>>> +    if (how & FLUSH_SYNC) {
>>> +            ret = rpc_wait_for_completion_task(task);
>>> +            if (ret == 0)
>>> +                    ret = task->tk_status;
>>> +    }
>>> +    rpc_put_task(task);
>>> +out:
>>> +    return ret;
>>> +}
>>> +
>>> +/*
>>> + * Set up the argument/result storage required for the RPC call.
>>> + */
>>> +static int nfs_write_rpcsetup(struct nfs_page *req,
>>> +            struct nfs_write_data *data,
>>> +            const struct rpc_call_ops *call_ops,
>>> +            unsigned int count, unsigned int offset,
>>> +            int how)
>>> +{
>>> +    struct inode *inode = req->wb_context->path.dentry->d_inode;
>>> +
>>>      /* Set up the RPC argument and reply structs
>>>       * NB: take care not to mess about with data->commit et al. */
>>>
>>>      data->req = req;
>>>      data->inode = inode = req->wb_context->path.dentry->d_inode;
>>> -    data->cred = msg.rpc_cred;
>>> +    data->cred = req->wb_context->cred;
>>>
>>>      data->args.fh     = NFS_FH(inode);
>>>      data->args.offset = req_offset(req) + offset;
>>> @@ -837,30 +870,7 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
>>>      data->res.verf    = &data->verf;
>>>      nfs_fattr_init(&data->fattr);
>>>
>>> -    /* Set up the initial task struct.  */
>>> -    NFS_PROTO(inode)->write_setup(data, &msg);
>>> -
>>> -    dprintk("NFS: %5u initiated write call "
>>> -            "(req %s/%lld, %u bytes @ offset %llu)\n",
>>> -            data->task.tk_pid,
>>> -            inode->i_sb->s_id,
>>> -            (long long)NFS_FILEID(inode),
>>> -            count,
>>> -            (unsigned long long)data->args.offset);
>>> -
>>> -    task = rpc_run_task(&task_setup_data);
>>> -    if (IS_ERR(task)) {
>>> -            ret = PTR_ERR(task);
>>> -            goto out;
>>> -    }
>>> -    if (how & FLUSH_SYNC) {
>>> -            ret = rpc_wait_for_completion_task(task);
>>> -            if (ret == 0)
>>> -                    ret = task->tk_status;
>>> -    }
>>> -    rpc_put_task(task);
>>> -out:
>>> -    return ret;
>>> +    return nfs_initiate_write(data, NFS_CLIENT(inode), call_ops, how);
>>> }
>>>
>>> /* If a nfs_flush_* function fails, it should remove reqs from @head and
>>> --
>>> 1.7.2.1
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

[PATCH 2/7] wave4: Send lseg down through nfs_write_rpcsetup into nfs_initiate_write

[Fred Isaman]

We grab the lseg sent in to the doio function and attach it to
each struct nfs_write_data created.  This is how the lseg will be
sent to the layout driver.

Signed-off-by: Fred Isaman <iisaman@netapp.com>
---
 fs/nfs/write.c          |    7 +++++--
 include/linux/nfs_xdr.h |    1 +
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 5604854..fa92fad 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -98,6 +98,7 @@ void nfs_writedata_free(struct nfs_write_data *p)
 static void nfs_writedata_release(struct nfs_write_data *wdata)
 {
 	put_nfs_open_context(wdata->args.context);
+	put_lseg(wdata->lseg);
 	nfs_writedata_free(wdata);
 }
 
@@ -840,6 +841,7 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
 		struct nfs_write_data *data,
 		const struct rpc_call_ops *call_ops,
 		unsigned int count, unsigned int offset,
+		struct pnfs_layout_segment *lseg,
 		int how)
 {
 	struct inode *inode = req->wb_context->path.dentry->d_inode;
@@ -850,6 +852,7 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
 	data->req = req;
 	data->inode = inode = req->wb_context->path.dentry->d_inode;
 	data->cred = req->wb_context->cred;
+	data->lseg = get_lseg(lseg);
 
 	data->args.fh     = NFS_FH(inode);
 	data->args.offset = req_offset(req) + offset;
@@ -930,7 +933,7 @@ static int nfs_flush_multi(struct inode *inode, struct list_head *head, unsigned
 		if (nbytes < wsize)
 			wsize = nbytes;
 		ret2 = nfs_write_rpcsetup(req, data, &nfs_write_partial_ops,
-				   wsize, offset, how);
+					  wsize, offset, lseg, how);
 		if (ret == 0)
 			ret = ret2;
 		offset += wsize;
@@ -978,7 +981,7 @@ static int nfs_flush_one(struct inode *inode, struct list_head *head, unsigned i
 	req = nfs_list_entry(data->pages.next);
 
 	/* Set up the argument struct */
-	return nfs_write_rpcsetup(req, data, &nfs_write_full_ops, count, 0, how);
+	return nfs_write_rpcsetup(req, data, &nfs_write_full_ops, count, 0, lseg, how);
  out_bad:
 	while (!list_empty(head)) {
 		req = nfs_list_entry(head->next);
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 4121c3e..d60ec32 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1038,6 +1038,7 @@ struct nfs_write_data {
 	unsigned int		npages;		/* Max length of pagevec */
 	struct nfs_writeargs	args;		/* argument struct */
 	struct nfs_writeres	res;		/* result struct */
+	struct pnfs_layout_segment *lseg;
 #ifdef CONFIG_NFS_V4
 	unsigned long		timestamp;	/* For lease renewal */
 #endif
-- 
1.7.2.1

[PATCH 3/7] wave4: remove GETATTR from ds write

[Fred Isaman]

Any WRITE compounds directed to a data server need to have the
GETATTR calls suppressed.

Signed-off-by: Fred Isaman <iisaman@netapp.com>
---
 fs/nfs/nfs4proc.c       |    6 +++++-
 fs/nfs/nfs4xdr.c        |    8 +++++---
 include/linux/nfs_xdr.h |    1 +
 3 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index f396853..a40a281 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -3144,7 +3144,11 @@ static void nfs4_proc_write_setup(struct nfs_write_data *data, struct rpc_messag
 {
 	struct nfs_server *server = NFS_SERVER(data->inode);
 
-	data->args.bitmask = server->cache_consistency_bitmask;
+	if (data->ds_clp) {
+		data->args.bitmask = NULL;
+		data->res.fattr = NULL;
+	} else
+		data->args.bitmask = server->cache_consistency_bitmask;
 	data->res.server = server;
 	data->timestamp   = jiffies;
 
diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index 2380c45..c35880c 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -2275,7 +2275,8 @@ static void nfs4_xdr_enc_write(struct rpc_rqst *req, struct xdr_stream *xdr,
 	encode_putfh(xdr, args->fh, &hdr);
 	encode_write(xdr, args, &hdr);
 	req->rq_snd_buf.flags |= XDRBUF_WRITE;
-	encode_getfattr(xdr, args->bitmask, &hdr);
+	if (args->bitmask)
+		encode_getfattr(xdr, args->bitmask, &hdr);
 	encode_nops(&hdr);
 }
 
@@ -5694,8 +5695,9 @@ static int nfs4_xdr_dec_write(struct rpc_rqst *rqstp, struct xdr_stream *xdr,
 	status = decode_write(xdr, res);
 	if (status)
 		goto out;
-	decode_getfattr(xdr, res->fattr, res->server,
-			!RPC_IS_ASYNC(rqstp->rq_task));
+	if (res->fattr)
+		decode_getfattr(xdr, res->fattr, res->server,
+				!RPC_IS_ASYNC(rqstp->rq_task));
 	if (!status)
 		status = res->count;
 out:
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index d60ec32..726ec2a 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1039,6 +1039,7 @@ struct nfs_write_data {
 	struct nfs_writeargs	args;		/* argument struct */
 	struct nfs_writeres	res;		/* result struct */
 	struct pnfs_layout_segment *lseg;
+	struct nfs_client	*ds_clp;	/* pNFS data server */
 #ifdef CONFIG_NFS_V4
 	unsigned long		timestamp;	/* For lease renewal */
 #endif
-- 
1.7.2.1

[PATCH 4/7] wave4: nfs: rearrange nfs4_write_done

[Fred Isaman]

Add hook that layout driver can use to take control here.

Signed-off-by: Fred Isaman <iisaman@netapp.com>
---
 fs/nfs/nfs4proc.c       |   14 ++++++++++----
 include/linux/nfs_xdr.h |    1 +
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index a40a281..3952f7c 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -3122,13 +3122,10 @@ void nfs4_reset_read(struct rpc_task *task, struct nfs_read_data *data)
 }
 EXPORT_SYMBOL_GPL(nfs4_reset_read);
 
-static int nfs4_write_done(struct rpc_task *task, struct nfs_write_data *data)
+static int nfs4_write_done_cb(struct rpc_task *task, struct nfs_write_data *data)
 {
 	struct inode *inode = data->inode;
 	
-	if (!nfs4_sequence_done(task, &data->res.seq_res))
-		return -EAGAIN;
-
 	if (nfs4_async_handle_error(task, NFS_SERVER(inode), data->args.context->state) == -EAGAIN) {
 		nfs_restart_rpc(task, NFS_SERVER(inode)->nfs_client);
 		return -EAGAIN;
@@ -3140,6 +3137,13 @@ static int nfs4_write_done(struct rpc_task *task, struct nfs_write_data *data)
 	return 0;
 }
 
+static int nfs4_write_done(struct rpc_task *task, struct nfs_write_data *data)
+{
+	if (!nfs4_sequence_done(task, &data->res.seq_res))
+		return -EAGAIN;
+	return data->write_done_cb(task, data);
+}
+
 static void nfs4_proc_write_setup(struct nfs_write_data *data, struct rpc_message *msg)
 {
 	struct nfs_server *server = NFS_SERVER(data->inode);
@@ -3149,6 +3153,8 @@ static void nfs4_proc_write_setup(struct nfs_write_data *data, struct rpc_messag
 		data->res.fattr = NULL;
 	} else
 		data->args.bitmask = server->cache_consistency_bitmask;
+	if (!data->write_done_cb)
+		data->write_done_cb = nfs4_write_done_cb;
 	data->res.server = server;
 	data->timestamp   = jiffies;
 
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 726ec2a..23feaf6 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1040,6 +1040,7 @@ struct nfs_write_data {
 	struct nfs_writeres	res;		/* result struct */
 	struct pnfs_layout_segment *lseg;
 	struct nfs_client	*ds_clp;	/* pNFS data server */
+	int (*write_done_cb) (struct rpc_task *task, struct nfs_write_data *data);
 #ifdef CONFIG_NFS_V4
 	unsigned long		timestamp;	/* For lease renewal */
 #endif
-- 
1.7.2.1

[PATCH 5/7] wave4: implement pnfs_try_to_write_data

[Fred Isaman]

From: The pNFS Team <linux-nfs@vger.kernel.org>

split from "pnfs_submit: generic write" and modified

Signed-off-by: Andy Adamson <andros@citi.umich.edu>
Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
Signed-off-by: Fred Isaman <iisaman@citi.umich.edu>
Signed-off-by: J. Bruce Fields <bfields@fieldses.org>
Signed-off-by: Mike Sager <sager@netapp.com>
Signed-off-by: Ricardo Labiaga <Ricardo.Labiaga@netapp.com>
Signed-off-by: Tao Guo <guotao@nrchpc.ac.cn>
Signed-off-by: Andy Adamson <andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
Signed-off-by: Fred Isaman <iisaman@netapp.com>
---
 fs/nfs/pnfs.c              |   24 ++++++++++++++++++++++++
 fs/nfs/pnfs.h              |   11 +++++++++++
 fs/nfs/write.c             |    4 ++++
 include/linux/nfs_iostat.h |    1 +
 include/linux/nfs_xdr.h    |    1 +
 5 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 912b1ff..9d2351a 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -892,6 +892,30 @@ pnfs_pageio_init_read(struct nfs_pageio_descriptor *pgio,
 	pnfs_set_pg_test(inode, pgio);
 }
 
+enum pnfs_try_status
+pnfs_try_to_write_data(struct nfs_write_data *wdata,
+			const struct rpc_call_ops *call_ops, int how)
+{
+	struct inode *inode = wdata->inode;
+	enum pnfs_try_status trypnfs;
+	struct nfs_server *nfss = NFS_SERVER(inode);
+
+	wdata->mds_ops = call_ops;
+
+	dprintk("%s: Writing ino:%lu %u@%llu (how %d)\n", __func__,
+		inode->i_ino, wdata->args.count, wdata->args.offset, how);
+
+	trypnfs = nfss->pnfs_curr_ld->write_pagelist(wdata, how);
+	if (trypnfs == PNFS_NOT_ATTEMPTED) {
+		wdata->lseg = NULL;
+		put_lseg(wdata->lseg);
+	} else
+		nfs_inc_stats(inode, NFSIOS_PNFS_WRITE);
+
+	dprintk("%s End (trypnfs:%d)\n", __func__, trypnfs);
+	return trypnfs;
+}
+
 /*
  * Call the appropriate parallel I/O subsystem read function.
  */
diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index d2d708a..50bcb1f 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -81,6 +81,8 @@ struct pnfs_layoutdriver_type {
 	 * I/O, else return PNFS_NOT_ATTEMPTED to fall back to normal NFS
 	 */
 	enum pnfs_try_status (*read_pagelist) (struct nfs_read_data *nfs_data);
+	enum pnfs_try_status
+	(*write_pagelist) (struct nfs_write_data *nfs_data, int how);
 };
 
 struct pnfs_layout_hdr {
@@ -172,6 +174,8 @@ pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx,
 		   enum pnfs_iomode access_type);
 void set_pnfs_layoutdriver(struct nfs_server *, u32 id);
 void unset_pnfs_layoutdriver(struct nfs_server *);
+enum pnfs_try_status pnfs_try_to_write_data(struct nfs_write_data *,
+					     const struct rpc_call_ops *, int);
 enum pnfs_try_status pnfs_try_to_read_data(struct nfs_read_data *,
 					    const struct rpc_call_ops *);
 void pnfs_pageio_init_read(struct nfs_pageio_descriptor *, struct inode *);
@@ -250,6 +254,13 @@ pnfs_try_to_read_data(struct nfs_read_data *data,
 	return PNFS_NOT_ATTEMPTED;
 }
 
+static inline enum pnfs_try_status
+pnfs_try_to_write_data(struct nfs_write_data *data,
+		       const struct rpc_call_ops *call_ops, int how)
+{
+	return PNFS_NOT_ATTEMPTED;
+}
+
 static inline bool
 pnfs_roc(struct inode *ino)
 {
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index fa92fad..24cc34d 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -873,6 +873,10 @@ static int nfs_write_rpcsetup(struct nfs_page *req,
 	data->res.verf    = &data->verf;
 	nfs_fattr_init(&data->fattr);
 
+	if (data->lseg &&
+	    (pnfs_try_to_write_data(data, call_ops, how) == PNFS_ATTEMPTED))
+		return 0;
+
 	return nfs_initiate_write(data, NFS_CLIENT(inode), call_ops, how);
 }
 
diff --git a/include/linux/nfs_iostat.h b/include/linux/nfs_iostat.h
index 37a1437..8866bb3 100644
--- a/include/linux/nfs_iostat.h
+++ b/include/linux/nfs_iostat.h
@@ -114,6 +114,7 @@ enum nfs_stat_eventcounters {
 	NFSIOS_SHORTWRITE,
 	NFSIOS_DELAY,
 	NFSIOS_PNFS_READ,
+	NFSIOS_PNFS_WRITE,
 	__NFSIOS_COUNTSMAX,
 };
 
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 23feaf6..9ff26f9 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1040,6 +1040,7 @@ struct nfs_write_data {
 	struct nfs_writeres	res;		/* result struct */
 	struct pnfs_layout_segment *lseg;
 	struct nfs_client	*ds_clp;	/* pNFS data server */
+	const struct rpc_call_ops *mds_ops;
 	int (*write_done_cb) (struct rpc_task *task, struct nfs_write_data *data);
 #ifdef CONFIG_NFS_V4
 	unsigned long		timestamp;	/* For lease renewal */
-- 
1.7.2.1

[PATCH 6/7] wave4: filelayout write

[Fred Isaman]

From: The pNFS Team <linux-nfs@vger.kernel.org>

Signed-off-by: Andy Adamson <andros@citi.umich.edu>
Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
Signed-off-by: Fred Isaman <iisaman@citi.umich.edu>
Signed-off-by: Mingyang Guo <guomingyang@nrchpc.ac.cn>
Signed-off-by: Oleg Drokin <green@linuxhacker.ru>
Signed-off-by: Ricardo Labiaga <Ricardo.Labiaga@netapp.com>
Tested-by: Guo Mingyang <guomingyang@nrchpc.ac.cn>
Signed-off-by: Andy Adamson <andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
Signed-off-by: Fred Isaman <iisaman@netapp.com>
---
 fs/nfs/internal.h       |    5 ++
 fs/nfs/nfs4filelayout.c |  103 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4proc.c       |   17 ++++++++
 fs/nfs/write.c          |    5 ++-
 include/linux/nfs_xdr.h |    1 +
 5 files changed, 130 insertions(+), 1 deletions(-)

diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index f69a322..34096a1 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -272,6 +272,10 @@ extern int nfs_initiate_read(struct nfs_read_data *data, struct rpc_clnt *clnt,
 extern void nfs_read_prepare(struct rpc_task *task, void *calldata);
 
 /* write.c */
+extern int nfs_initiate_write(struct nfs_write_data *data,
+			      struct rpc_clnt *clnt,
+			      const struct rpc_call_ops *call_ops,
+			      int how);
 extern void nfs_write_prepare(struct rpc_task *task, void *calldata);
 #ifdef CONFIG_MIGRATION
 extern int nfs_migrate_page(struct address_space *,
@@ -282,6 +286,7 @@ extern int nfs_migrate_page(struct address_space *,
 
 /* nfs4proc.c */
 extern void nfs4_reset_read(struct rpc_task *task, struct nfs_read_data *data);
+extern void nfs4_reset_write(struct rpc_task *task, struct nfs_write_data *data);
 extern int _nfs4_call_sync(struct nfs_server *server,
 			   struct rpc_message *msg,
 			   struct nfs4_sequence_args *args,
diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c
index 80e7a66..be4af22 100644
--- a/fs/nfs/nfs4filelayout.c
+++ b/fs/nfs/nfs4filelayout.c
@@ -208,12 +208,69 @@ static void filelayout_read_release(void *data)
 	rdata->mds_ops->rpc_release(data);
 }
 
+static int filelayout_write_done_cb(struct rpc_task *task,
+				struct nfs_write_data *data)
+{
+	int reset = 0;
+
+	if (filelayout_async_handle_error(task, data->args.context->state,
+					  data->ds_clp, &reset) == -EAGAIN) {
+		struct nfs_client *clp;
+
+		dprintk("%s calling restart ds_clp %p ds_clp->cl_session %p\n",
+			__func__, data->ds_clp, data->ds_clp->cl_session);
+		if (reset) {
+			nfs4_reset_write(task, data);
+			filelayout_set_lo_fail(data->lseg);
+			clp = NFS_SERVER(data->inode)->nfs_client;
+		} else
+			clp = data->ds_clp;
+		nfs_restart_rpc(task, clp);
+		return -EAGAIN;
+	}
+
+	return 0;
+}
+
+static void filelayout_write_prepare(struct rpc_task *task, void *data)
+{
+	struct nfs_write_data *wdata = (struct nfs_write_data *)data;
+
+	if (nfs41_setup_sequence(wdata->ds_clp->cl_session,
+				&wdata->args.seq_args, &wdata->res.seq_res,
+				0, task))
+		return;
+
+	rpc_call_start(task);
+}
+
+static void filelayout_write_call_done(struct rpc_task *task, void *data)
+{
+	struct nfs_write_data *wdata = (struct nfs_write_data *)data;
+
+	/* Note this may cause RPC to be resent */
+	wdata->mds_ops->rpc_call_done(task, data);
+}
+
+static void filelayout_write_release(void *data)
+{
+	struct nfs_write_data *wdata = (struct nfs_write_data *)data;
+
+	wdata->mds_ops->rpc_release(data);
+}
+
 struct rpc_call_ops filelayout_read_call_ops = {
 	.rpc_call_prepare = filelayout_read_prepare,
 	.rpc_call_done = filelayout_read_call_done,
 	.rpc_release = filelayout_read_release,
 };
 
+struct rpc_call_ops filelayout_write_call_ops = {
+	.rpc_call_prepare = filelayout_write_prepare,
+	.rpc_call_done = filelayout_write_call_done,
+	.rpc_release = filelayout_write_release,
+};
+
 static enum pnfs_try_status
 filelayout_read_pagelist(struct nfs_read_data *data)
 {
@@ -254,6 +311,51 @@ filelayout_read_pagelist(struct nfs_read_data *data)
 	return PNFS_ATTEMPTED;
 }
 
+/* Perform async writes. */
+static enum pnfs_try_status
+filelayout_write_pagelist(struct nfs_write_data *data, int sync)
+{
+	struct pnfs_layout_segment *lseg = data->lseg;
+	struct nfs4_pnfs_ds *ds;
+	loff_t offset = data->args.offset;
+	u32 idx;
+	struct nfs_fh *fh;
+
+	/* Retrieve the correct rpc_client for the byte range */
+	idx = nfs4_fl_calc_ds_index(lseg, offset);
+	ds = nfs4_fl_prepare_ds(lseg, idx);
+	if (!ds) {
+		printk(KERN_ERR "%s: prepare_ds failed, use MDS\n", __func__);
+		set_bit(lo_fail_bit(IOMODE_RW), &lseg->pls_layout->plh_flags);
+		set_bit(lo_fail_bit(IOMODE_READ), &lseg->pls_layout->plh_flags);
+		return PNFS_NOT_ATTEMPTED;
+	}
+	dprintk("%s ino %lu sync %d req %Zu@%llu DS:%x:%hu\n", __func__,
+		data->inode->i_ino, sync, (size_t) data->args.count, offset,
+		ntohl(ds->ds_ip_addr), ntohs(ds->ds_port));
+
+	/* We can't handle commit to ds yet */
+	if (!FILELAYOUT_LSEG(lseg)->commit_through_mds)
+		data->args.stable = NFS_FILE_SYNC;
+
+	data->write_done_cb = filelayout_write_done_cb;
+	data->ds_clp = ds->ds_clp;
+	fh = nfs4_fl_select_ds_fh(lseg, offset);
+	if (fh)
+		data->args.fh = fh;
+	/*
+	 * Get the file offset on the dserver. Set the write offset to
+	 * this offset and save the original offset.
+	 */
+	data->args.offset = filelayout_get_dserver_offset(lseg, offset);
+	data->mds_offset = offset;
+
+	/* Perform an asynchronous write */
+	nfs_initiate_write(data, ds->ds_clp->cl_rpcclient,
+			   &filelayout_write_call_ops, sync);
+	return PNFS_ATTEMPTED;
+}
+
 /*
  * filelayout_check_layout()
  *
@@ -475,6 +577,7 @@ static struct pnfs_layoutdriver_type filelayout_type = {
 	.free_lseg               = filelayout_free_lseg,
 	.pg_test                 = filelayout_pg_test,
 	.read_pagelist           = filelayout_read_pagelist,
+	.write_pagelist          = filelayout_write_pagelist,
 };
 
 static int __init nfs4filelayout_init(void)
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 3952f7c..6cce678 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -3144,6 +3144,23 @@ static int nfs4_write_done(struct rpc_task *task, struct nfs_write_data *data)
 	return data->write_done_cb(task, data);
 }
 
+/* Reset the the nfs_write_data to send the write to the MDS. */
+void nfs4_reset_write(struct rpc_task *task, struct nfs_write_data *data)
+{
+	dprintk("%s Reset task for i/o through \n", __func__);
+	put_lseg(data->lseg);
+	data->lseg          = NULL;
+	data->ds_clp        = NULL;
+	data->write_done_cb = nfs4_write_done_cb;
+	data->args.fh       = NFS_FH(data->inode);
+	data->args.bitmask  = data->res.server->cache_consistency_bitmask;
+	data->args.offset   = data->mds_offset;
+	data->res.fattr     = &data->fattr;
+	task->tk_ops        = data->mds_ops;
+	rpc_task_reset_client(task, NFS_CLIENT(data->inode));
+}
+EXPORT_SYMBOL_GPL(nfs4_reset_write);
+
 static void nfs4_proc_write_setup(struct nfs_write_data *data, struct rpc_message *msg)
 {
 	struct nfs_server *server = NFS_SERVER(data->inode);
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 24cc34d..5757b06 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -783,7 +783,7 @@ static int flush_task_priority(int how)
 	return RPC_PRIORITY_NORMAL;
 }
 
-static int nfs_initiate_write(struct nfs_write_data *data,
+int nfs_initiate_write(struct nfs_write_data *data,
 		       struct rpc_clnt *clnt,
 		       const struct rpc_call_ops *call_ops,
 		       int how)
@@ -833,6 +833,7 @@ static int nfs_initiate_write(struct nfs_write_data *data,
 out:
 	return ret;
 }
+EXPORT_SYMBOL(nfs_initiate_write);
 
 /*
  * Set up the argument/result storage required for the RPC call.
@@ -1186,6 +1187,7 @@ int nfs_writeback_done(struct rpc_task *task, struct nfs_write_data *data)
 		 */
 		static unsigned long    complain;
 
+		/* Note this will print the MDS for a DS write */
 		if (time_before(complain, jiffies)) {
 			dprintk("NFS:       faulty NFS server %s:"
 				" (committed = %d) != (stable = %d)\n",
@@ -1206,6 +1208,7 @@ int nfs_writeback_done(struct rpc_task *task, struct nfs_write_data *data)
 			/* Was this an NFSv2 write or an NFSv3 stable write? */
 			if (resp->verf->committed != NFS_UNSTABLE) {
 				/* Resend from where the server left off */
+				data->mds_offset += resp->count;
 				argp->offset += resp->count;
 				argp->pgbase += resp->count;
 				argp->count -= resp->count;
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 9ff26f9..a045d12 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1045,6 +1045,7 @@ struct nfs_write_data {
 #ifdef CONFIG_NFS_V4
 	unsigned long		timestamp;	/* For lease renewal */
 #endif
+	__u64			mds_offset;	/* Filelayout dense stripe */
 	struct page		*page_array[NFS_PAGEVEC_SIZE];
 };
 
-- 
1.7.2.1

[PATCH 7/7] wave4: nfs_pageio_do_add_request needs to distinguish between read and write

[Fred Isaman]

This is so that it can send the appropriate mode in LAYOUTGET.

Signed-off-by: Fred Isaman <iisaman@netapp.com>
---
 fs/nfs/pagelist.c        |    7 +++----
 fs/nfs/pnfs.c            |    8 ++++++++
 fs/nfs/pnfs.h            |    7 +++++++
 fs/nfs/write.c           |    2 +-
 include/linux/nfs_page.h |    1 +
 5 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index f0aa20b..0518d62a 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -299,10 +299,9 @@ static int nfs_pageio_do_add_request(struct nfs_pageio_descriptor *desc,
 	} else {
 		put_lseg(desc->pg_lseg);
 		desc->pg_base = req->wb_pgbase;
-		if (desc->pg_test)
-			desc->pg_lseg = pnfs_update_layout(desc->pg_inode,
-							   req->wb_context,
-							   IOMODE_READ);
+		desc->pg_lseg = pnfs_update_layout(desc->pg_inode,
+						   req->wb_context,
+						   desc->pg_iomode);
 	}
 	nfs_list_remove_request(req);
 	nfs_list_add_request(req, &desc->pg_list);
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 9d2351a..a2ee472 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -889,6 +889,14 @@ void
 pnfs_pageio_init_read(struct nfs_pageio_descriptor *pgio,
 		  struct inode *inode)
 {
+	pgio->pg_iomode = IOMODE_READ;
+	pnfs_set_pg_test(inode, pgio);
+}
+
+void
+pnfs_pageio_init_write(struct nfs_pageio_descriptor *pgio, struct inode *inode)
+{
+	pgio->pg_iomode = IOMODE_RW;
 	pnfs_set_pg_test(inode, pgio);
 }
 
diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index 50bcb1f..16f9fa5 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -179,6 +179,7 @@ enum pnfs_try_status pnfs_try_to_write_data(struct nfs_write_data *,
 enum pnfs_try_status pnfs_try_to_read_data(struct nfs_read_data *,
 					    const struct rpc_call_ops *);
 void pnfs_pageio_init_read(struct nfs_pageio_descriptor *, struct inode *);
+void pnfs_pageio_init_write(struct nfs_pageio_descriptor *, struct inode *);
 int pnfs_layout_process(struct nfs4_layoutget *lgp);
 void pnfs_free_lseg_list(struct list_head *tmp_list);
 void pnfs_destroy_layout(struct nfs_inode *);
@@ -297,6 +298,12 @@ pnfs_pageio_init_read(struct nfs_pageio_descriptor *pgio, struct inode *ino)
 	pgio->pg_test = NULL;
 }
 
+static inline void
+pnfs_pageio_init_write(struct nfs_pageio_descriptor *pgio, struct inode *ino)
+{
+	pgio->pg_test = NULL;
+}
+
 #endif /* CONFIG_NFS_V4_1 */
 
 #endif /* FS_NFS_PNFS_H */
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 5757b06..4c21045 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -1001,7 +1001,7 @@ static void nfs_pageio_init_write(struct nfs_pageio_descriptor *pgio,
 {
 	size_t wsize = NFS_SERVER(inode)->wsize;
 
-	pgio->pg_test = NULL;
+	pnfs_pageio_init_write(pgio, inode);
 
 	if (wsize < PAGE_CACHE_SIZE)
 		nfs_pageio_init(pgio, inode, nfs_flush_multi, wsize, ioflags);
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index ba88ff4..fd57be2 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -63,6 +63,7 @@ struct nfs_pageio_descriptor {
 	int 			pg_ioflags;
 	int			pg_error;
 	struct pnfs_layout_segment *pg_lseg;
+	int			pg_iomode;
 	int			(*pg_test)(struct nfs_pageio_descriptor *, struct nfs_page *, struct nfs_page *);
 };
 
-- 
1.7.2.1

pNFS files server?

[Daniel.Muntz]

Is anyone still working on a freely-available pNFS files server for Linux?  I've had a couple of inquiries recently about spNFS, and would like to be able to point people to recent work.  Last I knew, Andy's GFS2-based pnfs files server work was the POR for delivering a server.  Is that work still going on, and if so, is there usable code I can point to?

  Thanks,
  Dan

Re: pNFS files server?

[Andy Adamson]

The GFS2 pnfs files server code has not moved forward for a while as we have all spent our time on the pnfs files client.  The pnfsd-files branch of benny's tree git://linux-nfs.org/~bhalevy/linux-pnfs.git has the latest code.

-->Andy

On Feb 11, 2011, at 5:15 PM, <Daniel.Muntz@emc.com> wrote:

> Is anyone still working on a freely-available pNFS files server for Linux?  I've had a couple of inquiries recently about spNFS, and would like to be able to point people to recent work.  Last I knew, Andy's GFS2-based pnfs files server work was the POR for delivering a server.  Is that work still going on, and if so, is there usable code I can point to?
> 
>  Thanks,
>  Dan--
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] pnfs wave 4 proposal

[Andy Adamson]

I don't see pnfs_initiate_write

-->Andy

On Feb 11, 2011, at 11:54 AM, Fred Isaman wrote:

> These apply on top of Andy's wave3-on-nfs-for-next-rev4.
> 
> These patches give write functionality to the pnfs file layout driver.
> The primary limitation is that we have not yet modified the commit
> code, so we can not yet handle COMMITs to the data server.  We get
> around this for the moment by sending any WRITE to a data server which
> does not have commit-to-mds set as a SYNCH write.
> 
> Fred
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] pnfs wave 4 proposal

[Fred Isaman]

On Fri, Feb 11, 2011 at 3:19 PM, Andy Adamson <andros@netapp.com> wrote:
> I don't see pnfs_initiate_write
>
> -->Andy
>

It is inlined at the end of nfs_rpc_setup, added in patch 5/7.

Fred