cephfs performance

cephfs performance

[sheng qiu]

hi,

I am evaluating the cephfs performance and see unreasonable
performance when multiple threads accessing same mount point.

there are 10k files (300MB each) in the fuse mount point, when
increase multiple fio threads accessing those 10k files, the
performance is not scaled and bounded to ~40MB/s.

the ceph cluster has 36 OSDs on three high end servers, each OSD
backed by a NVMe drive. Theres's another servers running one MDS
process and monitors.

if pushing IO via fio rbd engine to the cluster directly, it has
pretty reasonable performance. Is there any special configuration i
should pay attention when setup cephfs or fuse client is the problem?
or i should use more MDS?

any suggestions would be appreciated.

Thanks,
Sheng

Re: cephfs performance

[Xiaoxi Chen]

we have seen fuse show much worse perofermance compared to kernel,
would you mind to test via kernel cephfs client( ubuntu 16.04 with 4.4
or rhel 7.2/7.3 should be good enough)

2017-07-18 13:27 GMT+08:00 sheng qiu <herbert1984106@gmail.com>:
> hi,
>
> I am evaluating the cephfs performance and see unreasonable
> performance when multiple threads accessing same mount point.
>
> there are 10k files (300MB each) in the fuse mount point, when
> increase multiple fio threads accessing those 10k files, the
> performance is not scaled and bounded to ~40MB/s.
>
> the ceph cluster has 36 OSDs on three high end servers, each OSD
> backed by a NVMe drive. Theres's another servers running one MDS
> process and monitors.
>
> if pushing IO via fio rbd engine to the cluster directly, it has
> pretty reasonable performance. Is there any special configuration i
> should pay attention when setup cephfs or fuse client is the problem?
> or i should use more MDS?
>
> any suggestions would be appreciated.
>
> Thanks,
> Sheng
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cephfs performance

[Patrick Donnelly]

On Mon, Jul 17, 2017 at 10:27 PM, sheng qiu <herbert1984106@gmail.com> wrote:
> hi,
>
> I am evaluating the cephfs performance and see unreasonable
> performance when multiple threads accessing same mount point.
>
> there are 10k files (300MB each) in the fuse mount point, when
> increase multiple fio threads accessing those 10k files, the
> performance is not scaled and bounded to ~40MB/s.
>
> the ceph cluster has 36 OSDs on three high end servers, each OSD
> backed by a NVMe drive. Theres's another servers running one MDS
> process and monitors.
>
> if pushing IO via fio rbd engine to the cluster directly, it has
> pretty reasonable performance. Is there any special configuration i
> should pay attention when setup cephfs or fuse client is the problem?
> or i should use more MDS?
>
> any suggestions would be appreciated.

What version are you running? Any special config option changes?

What I/O engines are you testing with fio? We need more details about
the testing to give you better feedback.

-- 
Patrick Donnelly

Re: cephfs performance

[sheng qiu]

ceph version is v11.2, no special config changes, just some regular
configurations for filestore backend.

Fio is using libaio, with direct=1, running 4KB block size.

I will try kernel client to see if there's any improvement.

Thanks,
Sheng

On Tue, Jul 18, 2017 at 8:58 AM, Patrick Donnelly <pdonnell@redhat.com> wrote:
> On Mon, Jul 17, 2017 at 10:27 PM, sheng qiu <herbert1984106@gmail.com> wrote:
>> hi,
>>
>> I am evaluating the cephfs performance and see unreasonable
>> performance when multiple threads accessing same mount point.
>>
>> there are 10k files (300MB each) in the fuse mount point, when
>> increase multiple fio threads accessing those 10k files, the
>> performance is not scaled and bounded to ~40MB/s.
>>
>> the ceph cluster has 36 OSDs on three high end servers, each OSD
>> backed by a NVMe drive. Theres's another servers running one MDS
>> process and monitors.
>>
>> if pushing IO via fio rbd engine to the cluster directly, it has
>> pretty reasonable performance. Is there any special configuration i
>> should pay attention when setup cephfs or fuse client is the problem?
>> or i should use more MDS?
>>
>> any suggestions would be appreciated.
>
> What version are you running? Any special config option changes?
>
> What I/O engines are you testing with fio? We need more details about
> the testing to give you better feedback.
>
> --
> Patrick Donnelly

Re: cephfs performance

[Patrick Donnelly]

On Tue, Jul 18, 2017 at 9:07 AM, sheng qiu <herbert1984106@gmail.com> wrote:
> ceph version is v11.2, no special config changes, just some regular
> configurations for filestore backend.
>
> Fio is using libaio, with direct=1, running 4KB block size.

Don't use direct I/O unless you want to benchmark synchronous I/O:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019478.html

-- 
Patrick Donnelly

Re: cephfs performance

[sheng qiu]

Hi Patrick,

thanks for your quick response.
just one concern, if disable direct I/O, multiple clients access same
file via different mount point on different machines, will it cause
data inconsistency?
for example, some client modify one file which has old data buffered
on the other client.

Thanks,
Sheng

On Tue, Jul 18, 2017 at 9:11 AM, Patrick Donnelly <pdonnell@redhat.com> wrote:
> On Tue, Jul 18, 2017 at 9:07 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>> ceph version is v11.2, no special config changes, just some regular
>> configurations for filestore backend.
>>
>> Fio is using libaio, with direct=1, running 4KB block size.
>
> Don't use direct I/O unless you want to benchmark synchronous I/O:
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019478.html
>
> --
> Patrick Donnelly

Re: cephfs performance

[Patrick Donnelly]

On Tue, Jul 18, 2017 at 9:17 AM, sheng qiu <herbert1984106@gmail.com> wrote:
> Hi Patrick,
>
> thanks for your quick response.
> just one concern, if disable direct I/O, multiple clients access same
> file via different mount point on different machines, will it cause
> data inconsistency?

No, but you'll still effectively have synchronous I/O because only one
writer will have write capabilities from the MDS.

-- 
Patrick Donnelly

Re: cephfs performance

[sheng qiu]

Hi Patrick,

just to clarify your previous reply to make sure i understood
correctly. Because consistency is very important to our applications.

so when multiple write clients or a mix of read and write clients
accessing same file, the previous cached data of this file on all
clients will be revoked.
And the following multi-clients' write or mixed read and write
accessing to that file will become synchronous.

is this what you mean? if not please clarify it.

Thanks,
Sheng

On Tue, Jul 18, 2017 at 9:41 AM, Patrick Donnelly <pdonnell@redhat.com> wrote:
> On Tue, Jul 18, 2017 at 9:17 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>> Hi Patrick,
>>
>> thanks for your quick response.
>> just one concern, if disable direct I/O, multiple clients access same
>> file via different mount point on different machines, will it cause
>> data inconsistency?
>
> No, but you'll still effectively have synchronous I/O because only one
> writer will have write capabilities from the MDS.
>
> --
> Patrick Donnelly

Re: cephfs performance

[Patrick Donnelly]

Hi Sheng,

Sorry I misspoke in my previous email. The clients may all have the
capability to read and write but none of them will be able to
**cache** reads/writes or **buffer** writes. They rely on synchronous
read/write with the OSDs for consistency/atomicity.

On Fri, Jul 21, 2017 at 9:19 AM, sheng qiu <herbert1984106@gmail.com> wrote:
> so when multiple write clients or a mix of read and write clients
> accessing same file, the previous cached data of this file on all
> clients will be revoked.

Yes.

> And the following multi-clients' write or mixed read and write
> accessing to that file will become synchronous.

Writes/reads are become synchronous, yes.

> On Tue, Jul 18, 2017 at 9:41 AM, Patrick Donnelly <pdonnell@redhat.com> wrote:
>> On Tue, Jul 18, 2017 at 9:17 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>>> Hi Patrick,
>>>
>>> thanks for your quick response.
>>> just one concern, if disable direct I/O, multiple clients access same
>>> file via different mount point on different machines, will it cause
>>> data inconsistency?
>>
>> No, but you'll still effectively have synchronous I/O because only one
>> writer will have write capabilities from the MDS.
>>
>> --
>> Patrick Donnelly



-- 
Patrick Donnelly

Re: cephfs performance

[陶冬冬]

Hi Patrick,

sorry to dig out this old email.
you said two clients may write same file at same time only if no any clients got “buffer/cache” capability.
here my question is:
the two client would got two different “mtime” of that inode, since they can make “Fw” cap dirty. 
how would mds behave when the two client flushes the dirty caps?

Regards,
Dongdong


> 在 2017年7月22日，上午2:55，Patrick Donnelly <pdonnell@redhat.com> 写道：
> 
> Hi Sheng,
> 
> Sorry I misspoke in my previous email. The clients may all have the
> capability to read and write but none of them will be able to
> **cache** reads/writes or **buffer** writes. They rely on synchronous
> read/write with the OSDs for consistency/atomicity.
> 
> On Fri, Jul 21, 2017 at 9:19 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>> so when multiple write clients or a mix of read and write clients
>> accessing same file, the previous cached data of this file on all
>> clients will be revoked.
> 
> Yes.
> 
>> And the following multi-clients' write or mixed read and write
>> accessing to that file will become synchronous.
> 
> Writes/reads are become synchronous, yes.
> 
>> On Tue, Jul 18, 2017 at 9:41 AM, Patrick Donnelly <pdonnell@redhat.com> wrote:
>>> On Tue, Jul 18, 2017 at 9:17 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>>>> Hi Patrick,
>>>> 
>>>> thanks for your quick response.
>>>> just one concern, if disable direct I/O, multiple clients access same
>>>> file via different mount point on different machines, will it cause
>>>> data inconsistency?
>>> 
>>> No, but you'll still effectively have synchronous I/O because only one
>>> writer will have write capabilities from the MDS.
>>> 
>>> --
>>> Patrick Donnelly
> 
> 
> 
> -- 
> Patrick Donnelly
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cephfs performance

[Yan, Zheng]

On Thu, Nov 2, 2017 at 5:48 PM, 陶冬冬 <tdd21151186@gmail.com> wrote:
> Hi Patrick,
>
> sorry to dig out this old email.
> you said two clients may write same file at same time only if no any clients got “buffer/cache” capability.
> here my question is:
> the two client would got two different “mtime” of that inode, since they can make “Fw” cap dirty.
> how would mds behave when the two client flushes the dirty caps?
>

when handling caps flush for multiple client,  mds never set mtime of
inode back to an old time



> Regards,
> Dongdong
>
>
>> 在 2017年7月22日，上午2:55，Patrick Donnelly <pdonnell@redhat.com> 写道：
>>
>> Hi Sheng,
>>
>> Sorry I misspoke in my previous email. The clients may all have the
>> capability to read and write but none of them will be able to
>> **cache** reads/writes or **buffer** writes. They rely on synchronous
>> read/write with the OSDs for consistency/atomicity.
>>
>> On Fri, Jul 21, 2017 at 9:19 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>>> so when multiple write clients or a mix of read and write clients
>>> accessing same file, the previous cached data of this file on all
>>> clients will be revoked.
>>
>> Yes.
>>
>>> And the following multi-clients' write or mixed read and write
>>> accessing to that file will become synchronous.
>>
>> Writes/reads are become synchronous, yes.
>>
>>> On Tue, Jul 18, 2017 at 9:41 AM, Patrick Donnelly <pdonnell@redhat.com> wrote:
>>>> On Tue, Jul 18, 2017 at 9:17 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>>>>> Hi Patrick,
>>>>>
>>>>> thanks for your quick response.
>>>>> just one concern, if disable direct I/O, multiple clients access same
>>>>> file via different mount point on different machines, will it cause
>>>>> data inconsistency?
>>>>
>>>> No, but you'll still effectively have synchronous I/O because only one
>>>> writer will have write capabilities from the MDS.
>>>>
>>>> --
>>>> Patrick Donnelly
>>
>>
>>
>> --
>> Patrick Donnelly
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>