multiple client read/write in cephfs

multiple client read/write in cephfs

[陶冬冬]

Dear Cephers,

I have one issue about cephfs, suppose we got below scenario:
client A write one file with interval1 (offset1, length1)
then client B read the same file with interval2 (offset2, length2).
but due to capability system controlled by mds,  client B have to wait until client A finish its write operation (including flushing its buffer to osd)
Even if interval1 and interval2 belongs to two different object of that file,  client B still  have to wait client A.

I’m wondering why mds doesn’t try to control the caps based on object rather than whole file? 

Thanks,
Dongdong

Re: multiple client read/write in cephfs

[Yan, Zheng]

On Mon, Oct 23, 2017 at 12:50 PM, 陶冬冬 <tdd21151186@gmail.com> wrote:
> Dear Cephers,
>
> I have one issue about cephfs, suppose we got below scenario:
> client A write one file with interval1 (offset1, length1)
> then client B read the same file with interval2 (offset2, length2).
> but due to capability system controlled by mds,  client B have to wait until client A finish its write operation (including flushing its buffer to osd)

Client B only need to wait until Client A finishes flush buffer to
osd. If Client B keeps the file descriptor open, writing to page cache
gets disabled on client A, reading from client B does not need to wait
for writing on client A (different object).

> Even if interval1 and interval2 belongs to two different object of that file,  client B still  have to wait client A.
>
> I’m wondering why mds doesn’t try to control the caps based on object rather than whole file?

because mds can't foresee which offset client want to read/write. the
overhead for issuing caps for each read/write syscall is too high.

>
> Thanks,
> Dongdong

Re: multiple client read/write in cephfs

[陶冬冬]

Thanks Zheng,  so, do you mean client B can have “Fs" cap and thle same time client A can have “Fw” for the same file ?
so that, client A and client B can write and read the same file at same time (different object) ?

also another extended question, does that mean client A and client B can both have the “Fw” for the same file same time, and also they can write the same file at same time (different object)  ? 

Dongdong



> 在 2017年10月23日，下午3:55，Yan, Zheng <ukernel@gmail.com> 写道：
> 
> On Mon, Oct 23, 2017 at 12:50 PM, 陶冬冬 <tdd21151186@gmail.com> wrote:
>> Dear Cephers,
>> 
>> I have one issue about cephfs, suppose we got below scenario:
>> client A write one file with interval1 (offset1, length1)
>> then client B read the same file with interval2 (offset2, length2).
>> but due to capability system controlled by mds,  client B have to wait until client A finish its write operation (including flushing its buffer to osd)
> 
> Client B only need to wait until Client A finishes flush buffer to
> osd. If Client B keeps the file descriptor open, writing to page cache
> gets disabled on client A, reading from client B does not need to wait
> for writing on client A (different object).
> 
>> Even if interval1 and interval2 belongs to two different object of that file,  client B still  have to wait client A.
>> 
>> I’m wondering why mds doesn’t try to control the caps based on object rather than whole file?
> 
> because mds can't foresee which offset client want to read/write. the
> overhead for issuing caps for each read/write syscall is too high.
> 
>> 
>> Thanks,
>> Dongdong

Re: multiple client read/write in cephfs

[Yan, Zheng]

On Mon, Oct 23, 2017 at 4:25 PM, 陶冬冬 <tdd21151186@gmail.com> wrote:
>
> Thanks Zheng,  so, do you mean client B can have “Fs" cap and thle same time client A can have “Fw” for the same file ?
> so that, client A and client B can write and read the same file at same time (different object) ?
>

Reading within file size only requires "Fr" cap.  If read reaches file
size and there is no "Fs", checking EOF requires an extra getattr
request.


> also another extended question, does that mean client A and client B can both have the “Fw” for the same file same time, and also they can write the same file at same time (different object)  ?
>
> Dongdong
>
>
>
>> 在 2017年10月23日，下午3:55，Yan, Zheng <ukernel@gmail.com> 写道：
>>
>> On Mon, Oct 23, 2017 at 12:50 PM, 陶冬冬 <tdd21151186@gmail.com> wrote:
>>> Dear Cephers,
>>>
>>> I have one issue about cephfs, suppose we got below scenario:
>>> client A write one file with interval1 (offset1, length1)
>>> then client B read the same file with interval2 (offset2, length2).
>>> but due to capability system controlled by mds,  client B have to wait until client A finish its write operation (including flushing its buffer to osd)
>>
>> Client B only need to wait until Client A finishes flush buffer to
>> osd. If Client B keeps the file descriptor open, writing to page cache
>> gets disabled on client A, reading from client B does not need to wait
>> for writing on client A (different object).
>>
>>> Even if interval1 and interval2 belongs to two different object of that file,  client B still  have to wait client A.
>>>
>>> I’m wondering why mds doesn’t try to control the caps based on object rather than whole file?
>>
>> because mds can't foresee which offset client want to read/write. the
>> overhead for issuing caps for each read/write syscall is too high.
>>
>>>
>>> Thanks,
>>> Dongdong
>