spnfs write performance issue

spnfs write performance issue

[Fu Liankun]

When copy a big file(about 500MB) to nfs server using pnfs, it cost
obvious longer time as compared with NFSv3 or NFSv4 protocol.

PNFS cost about 300-550s
NFSv3 cost about 49s
NFSv4 cost about 49s

My test environment as following:

MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
ipaddr2:192.168.1.19

192.168.1.100 <-------- NFS4.1 connection --------->192.168.1.101
192.168.0.100 <-------- NFS4.1 connection --------->192.168.0.102
192.168.0.19  <-------- NFS4.1 connection --------->192.168.0.100

Dunring the write process, the client output the message:
FS-Cache: Loaded
FS-Cache: Netfs 'nfs' registered for caching
nfs4filelayout_init: NFSv4 File Layout Driver Registering...
nfs: server 192.168.1.101 not responding, timed out

The message "nfs: server 192.168.1.101 not responding, timed out" was
output during the COMMIT request.

Who can tell me the reason why pnfs cost so long time.

Re: spnfs write performance issue

[J. Bruce Fields]

On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
> When copy a big file(about 500MB) to nfs server using pnfs, it cost
> obvious longer time as compared with NFSv3 or NFSv4 protocol.
> 
> PNFS cost about 300-550s
> NFSv3 cost about 49s
> NFSv4 cost about 49s
> 
> My test environment as following:
> 
> MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
> DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
> DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
> client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,

What kind of server are you using?

--b.

> ipaddr2:192.168.1.19
> 
> 192.168.1.100 <-------- NFS4.1 connection --------->192.168.1.101
> 192.168.0.100 <-------- NFS4.1 connection --------->192.168.0.102
> 192.168.0.19  <-------- NFS4.1 connection --------->192.168.0.100
> 
> Dunring the write process, the client output the message:
> FS-Cache: Loaded
> FS-Cache: Netfs 'nfs' registered for caching
> nfs4filelayout_init: NFSv4 File Layout Driver Registering...
> nfs: server 192.168.1.101 not responding, timed out
> 
> The message "nfs: server 192.168.1.101 not responding, timed out" was
> output during the COMMIT request.
> 
> Who can tell me the reason why pnfs cost so long time.
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: spnfs write performance issue

[Fu Liankun]

J. Bruce Fields 写道:
> On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
>> When copy a big file(about 500MB) to nfs server using pnfs, it cost
>> obvious longer time as compared with NFSv3 or NFSv4 protocol.
>>
>> PNFS cost about 300-550s
>> NFSv3 cost about 49s
>> NFSv4 cost about 49s
>>
>> My test environment as following:
>>
>> MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
>> DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
>> DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
>> client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
> 
> What kind of server are you using?

Sorry for late response.

Fedora14 + 3.1.0-rc8-pnfs

--b

> 
> --b.
> 
>> ipaddr2:192.168.1.19
>>
>> 192.168.1.100 <-------- NFS4.1 connection --------->192.168.1.101
>> 192.168.0.100 <-------- NFS4.1 connection --------->192.168.0.102
>> 192.168.0.19  <-------- NFS4.1 connection --------->192.168.0.100
>>
>> Dunring the write process, the client output the message:
>> FS-Cache: Loaded
>> FS-Cache: Netfs 'nfs' registered for caching
>> nfs4filelayout_init: NFSv4 File Layout Driver Registering...
>> nfs: server 192.168.1.101 not responding, timed out
>>
>> The message "nfs: server 192.168.1.101 not responding, timed out" was
>> output during the COMMIT request.
>>
>> Who can tell me the reason why pnfs cost so long time.
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: spnfs write performance issue

[J. Bruce Fields]

On Mon, Feb 27, 2012 at 02:27:24PM +0800, Fu Liankun wrote:
> J. Bruce Fields 写道:
> >On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
> >>When copy a big file(about 500MB) to nfs server using pnfs, it cost
> >>obvious longer time as compared with NFSv3 or NFSv4 protocol.
> >>
> >>PNFS cost about 300-550s
> >>NFSv3 cost about 49s
> >>NFSv4 cost about 49s
> >>
> >>My test environment as following:
> >>
> >>MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
> >>DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
> >>DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
> >>client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
> >
> >What kind of server are you using?
> 
> Sorry for late response.
> 
> Fedora14 + 3.1.0-rc8-pnfs

And you're exporting gfs2?

--b.

> 
> --b
> 
> >
> >--b.
> >
> >>ipaddr2:192.168.1.19
> >>
> >>192.168.1.100 <-------- NFS4.1 connection --------->192.168.1.101
> >>192.168.0.100 <-------- NFS4.1 connection --------->192.168.0.102
> >>192.168.0.19  <-------- NFS4.1 connection --------->192.168.0.100
> >>
> >>Dunring the write process, the client output the message:
> >>FS-Cache: Loaded
> >>FS-Cache: Netfs 'nfs' registered for caching
> >>nfs4filelayout_init: NFSv4 File Layout Driver Registering...
> >>nfs: server 192.168.1.101 not responding, timed out
> >>
> >>The message "nfs: server 192.168.1.101 not responding, timed out" was
> >>output during the COMMIT request.
> >>
> >>Who can tell me the reason why pnfs cost so long time.
> >>
> >>
> >>
> >>--
> >>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>the body of a message to majordomo@vger.kernel.org
> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >--
> >To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 

Re: spnfs write performance issue

[Fu Liankun]

J. Bruce Fields 写道:
> On Mon, Feb 27, 2012 at 02:27:24PM +0800, Fu Liankun wrote:
>> J. Bruce Fields 写道:
>>> On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
>>>> When copy a big file(about 500MB) to nfs server using pnfs, it cost
>>>> obvious longer time as compared with NFSv3 or NFSv4 protocol.
>>>>
>>>> PNFS cost about 300-550s
>>>> NFSv3 cost about 49s
>>>> NFSv4 cost about 49s
>>>>
>>>> My test environment as following:
>>>>
>>>> MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
>>>> DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
>>>> DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
>>>> client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
>>> What kind of server are you using?
>> Sorry for late response.
>>
>> Fedora14 + 3.1.0-rc8-pnfs
> 
> And you're exporting gfs2?

No. I'm exporting ext4.
Does the file system type would influence the result?

--b

> 
> --b.
> 
>> --b
>>
>>> --b.
>>>
>>>> ipaddr2:192.168.1.19
>>>>
>>>> 192.168.1.100 <-------- NFS4.1 connection --------->192.168.1.101
>>>> 192.168.0.100 <-------- NFS4.1 connection --------->192.168.0.102
>>>> 192.168.0.19  <-------- NFS4.1 connection --------->192.168.0.100
>>>>
>>>> Dunring the write process, the client output the message:
>>>> FS-Cache: Loaded
>>>> FS-Cache: Netfs 'nfs' registered for caching
>>>> nfs4filelayout_init: NFSv4 File Layout Driver Registering...
>>>> nfs: server 192.168.1.101 not responding, timed out
>>>>
>>>> The message "nfs: server 192.168.1.101 not responding, timed out" was
>>>> output during the COMMIT request.
>>>>
>>>> Who can tell me the reason why pnfs cost so long time.
>>>>
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
----
Best wishes
Fu Liankun

Re: spnfs write performance issue

[J. Bruce Fields]

On Tue, Feb 28, 2012 at 09:32:26AM +0800, Fu Liankun wrote:
> J. Bruce Fields 写道:
> >On Mon, Feb 27, 2012 at 02:27:24PM +0800, Fu Liankun wrote:
> >>J. Bruce Fields 写道:
> >>>On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
> >>>>When copy a big file(about 500MB) to nfs server using pnfs, it cost
> >>>>obvious longer time as compared with NFSv3 or NFSv4 protocol.
> >>>>
> >>>>PNFS cost about 300-550s
> >>>>NFSv3 cost about 49s
> >>>>NFSv4 cost about 49s
> >>>>
> >>>>My test environment as following:
> >>>>
> >>>>MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
> >>>>DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
> >>>>DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
> >>>>client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
> >>>What kind of server are you using?
> >>Sorry for late response.
> >>
> >>Fedora14 + 3.1.0-rc8-pnfs
> >
> >And you're exporting gfs2?
> 
> No. I'm exporting ext4.
> Does the file system type would influence the result?

So you're using local-pnfs?  Or spnfs?  What setup instructions did you
follow?

--b.

Re: spnfs write performance issue

[Fu Liankun]

J. Bruce Fields 写道:
> On Tue, Feb 28, 2012 at 09:32:26AM +0800, Fu Liankun wrote:
>> J. Bruce Fields 写道:
>>> On Mon, Feb 27, 2012 at 02:27:24PM +0800, Fu Liankun wrote:
>>>> J. Bruce Fields 写道:
>>>>> On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
>>>>>> When copy a big file(about 500MB) to nfs server using pnfs, it cost
>>>>>> obvious longer time as compared with NFSv3 or NFSv4 protocol.
>>>>>>
>>>>>> PNFS cost about 300-550s
>>>>>> NFSv3 cost about 49s
>>>>>> NFSv4 cost about 49s
>>>>>>
>>>>>> My test environment as following:
>>>>>>
>>>>>> MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
>>>>>> DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
>>>>>> DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
>>>>>> client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
>>>>> What kind of server are you using?
>>>> Sorry for late response.
>>>>
>>>> Fedora14 + 3.1.0-rc8-pnfs
>>> And you're exporting gfs2?
>> No. I'm exporting ext4.
>> Does the file system type would influence the result?
> 
> So you're using local-pnfs?  Or spnfs?  What setup instructions did you
> follow?

I'm using spnfs.

--b

> 
> --b.
> 

Re: spnfs write performance issue

[J. Bruce Fields]

On Tue, Feb 28, 2012 at 09:48:18AM +0800, Fu Liankun wrote:
> J. Bruce Fields 写道:
> >On Tue, Feb 28, 2012 at 09:32:26AM +0800, Fu Liankun wrote:
> >>J. Bruce Fields 写道:
> >>>On Mon, Feb 27, 2012 at 02:27:24PM +0800, Fu Liankun wrote:
> >>>>J. Bruce Fields 写道:
> >>>>>On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
> >>>>>>When copy a big file(about 500MB) to nfs server using pnfs, it cost
> >>>>>>obvious longer time as compared with NFSv3 or NFSv4 protocol.
> >>>>>>
> >>>>>>PNFS cost about 300-550s
> >>>>>>NFSv3 cost about 49s
> >>>>>>NFSv4 cost about 49s
> >>>>>>
> >>>>>>My test environment as following:
> >>>>>>
> >>>>>>MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
> >>>>>>DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
> >>>>>>DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
> >>>>>>client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
> >>>>>What kind of server are you using?
> >>>>Sorry for late response.
> >>>>
> >>>>Fedora14 + 3.1.0-rc8-pnfs
> >>>And you're exporting gfs2?
> >>No. I'm exporting ext4.
> >>Does the file system type would influence the result?
> >
> >So you're using local-pnfs?  Or spnfs?  What setup instructions did you
> >follow?
> 
> I'm using spnfs.

OK, got it.

I'm not sure if anyone here can talk about the performance of spnfs.
It's not really maintained as far as I know.

So, you're the expert.

--b.

Re: spnfs write performance issue

[Fu Liankun]

J. Bruce Fields 写道:
> On Tue, Feb 28, 2012 at 09:48:18AM +0800, Fu Liankun wrote:
>> J. Bruce Fields 写道:
>>> On Tue, Feb 28, 2012 at 09:32:26AM +0800, Fu Liankun wrote:
>>>> J. Bruce Fields 写道:
>>>>> On Mon, Feb 27, 2012 at 02:27:24PM +0800, Fu Liankun wrote:
>>>>>> J. Bruce Fields 写道:
>>>>>>> On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
>>>>>>>> When copy a big file(about 500MB) to nfs server using pnfs, it cost
>>>>>>>> obvious longer time as compared with NFSv3 or NFSv4 protocol.
>>>>>>>>
>>>>>>>> PNFS cost about 300-550s
>>>>>>>> NFSv3 cost about 49s
>>>>>>>> NFSv4 cost about 49s
>>>>>>>>
>>>>>>>> My test environment as following:
>>>>>>>>
>>>>>>>> MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
>>>>>>>> DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
>>>>>>>> DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
>>>>>>>> client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
>>>>>>> What kind of server are you using?
>>>>>> Sorry for late response.
>>>>>>
>>>>>> Fedora14 + 3.1.0-rc8-pnfs
>>>>> And you're exporting gfs2?
>>>> No. I'm exporting ext4.
>>>> Does the file system type would influence the result?
>>> So you're using local-pnfs?  Or spnfs?  What setup instructions did you
>>> follow?
>> I'm using spnfs.
> 
> OK, got it.
> 
> I'm not sure if anyone here can talk about the performance of spnfs.
> It's not really maintained as far as I know.
>

Got it.
I will keep on investigating the issue.

--b


> So, you're the expert.
> 





-- 
----
Best wishes
Fu Liankun

Re: spnfs write performance issue

[Benny Halevy]

On 2012-02-28 03:53, J. Bruce Fields wrote:
> On Tue, Feb 28, 2012 at 09:48:18AM +0800, Fu Liankun wrote:
>> J. Bruce Fields 写道:
>>> On Tue, Feb 28, 2012 at 09:32:26AM +0800, Fu Liankun wrote:
>>>> J. Bruce Fields 写道:
>>>>> On Mon, Feb 27, 2012 at 02:27:24PM +0800, Fu Liankun wrote:
>>>>>> J. Bruce Fields 写道:
>>>>>>> On Wed, Feb 22, 2012 at 06:16:17PM +0800, Fu Liankun wrote:
>>>>>>>> When copy a big file(about 500MB) to nfs server using pnfs, it cost
>>>>>>>> obvious longer time as compared with NFSv3 or NFSv4 protocol.
>>>>>>>>
>>>>>>>> PNFS cost about 300-550s
>>>>>>>> NFSv3 cost about 49s
>>>>>>>> NFSv4 cost about 49s
>>>>>>>>
>>>>>>>> My test environment as following:
>>>>>>>>
>>>>>>>> MDS: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.100, ipaddr2:192.168.1.100
>>>>>>>> DS1: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.101, ipaddr2:192.168.1.101
>>>>>>>> DS2: 3.1.0-rc8-pnfs   ipaddr1: 192.168.0.102, ipaddr2:192.168.1.102
>>>>>>>> client: RHEL6.2GA(2.6.32-220.el6.i686)    ipaddr1: 192.168.0.19,
>>>>>>> What kind of server are you using?
>>>>>> Sorry for late response.
>>>>>>
>>>>>> Fedora14 + 3.1.0-rc8-pnfs
>>>>> And you're exporting gfs2?
>>>> No. I'm exporting ext4.
>>>> Does the file system type would influence the result?
>>>
>>> So you're using local-pnfs?  Or spnfs?  What setup instructions did you
>>> follow?
>>
>> I'm using spnfs.
> 
> OK, got it.
> 
> I'm not sure if anyone here can talk about the performance of spnfs.
> It's not really maintained as far as I know.

True.  spnfs is unsupported and not maintained by anybody.
Sorry.

Benny

> 
> So, you're the expert.
> 
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html