The cache problem of re-export NFS test

The cache problem of re-export NFS test

[Sylar Shen]

Hi developers,
I know that the following question may not be relative to Ceph but
this confuses me a lot.
I am testing the function of re-exporting nfs by command "dd" without
using buffer.
That means I would like my data to be written to Ceph DFS directly
without passing by the buffer of client or nfs-client.
Here is my test architecture. Hope you could understand what I am
trying to express.....:p
------------------------------------------------------------------------------
                   nfs                             ceph protocol
Client <------------>Ceph-Client <-------------------> Ceph DFS
------------------------------------------------------------------------------
I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
count=10000 oflag=dsync"
My purpose was to write the data from the client to Ceph DFS through
Ceph-client.
Ceph DFS was mounted by Ceph-client using Ceph protocol and
Ceph-client was mounted by client using nfs protocol.
I know that my dd command with argument oflag=dsync would write the
data to Ceph-client without using buffer on the client.
But I want my Ceph-client could also just pass the data to Ceph DFS
without using its buffer.
So.....is there any way I can run my scenario?
Or in other words, is there any method I can assure that Ceph-client
won't do the buffer task itself?
Thank you~~
-- 
Best Regards,
Sylar Shen

Re: The cache problem of re-export NFS test

[Sage Weil]

On Tue, 12 Apr 2011, Sylar Shen wrote:
> Hi developers,
> I know that the following question may not be relative to Ceph but
> this confuses me a lot.
> I am testing the function of re-exporting nfs by command "dd" without
> using buffer.
> That means I would like my data to be written to Ceph DFS directly
> without passing by the buffer of client or nfs-client.
> Here is my test architecture. Hope you could understand what I am
> trying to express.....:p
> ------------------------------------------------------------------------------
>                    nfs                             ceph protocol
> Client <------------>Ceph-Client <-------------------> Ceph DFS
> ------------------------------------------------------------------------------
> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
> count=10000 oflag=dsync"
> My purpose was to write the data from the client to Ceph DFS through
> Ceph-client.
> Ceph DFS was mounted by Ceph-client using Ceph protocol and
> Ceph-client was mounted by client using nfs protocol.
> I know that my dd command with argument oflag=dsync would write the
> data to Ceph-client without using buffer on the client.
> But I want my Ceph-client could also just pass the data to Ceph DFS
> without using its buffer.
> So.....is there any way I can run my scenario?
> Or in other words, is there any method I can assure that Ceph-client
> won't do the buffer task itself?

I think what you're looking for is the 'sync' option in /etc/exports on 
the ceph-client machine.  See 'man 5 exports'.  That will slow things down 
considerably but will give you the consistency and safety you're looking 
for.

sage

Re: The cache problem of re-export NFS test

[Sylar Shen]

Hi Sage,
Thanks for your reply. I really appreciated.
I tried what you suggested to set the argument "sync" in /etc/exports.
I also checked "man 5 exports" and it showed that sync means reply to
requests only after the changes have been committed to stable storage.
It didn't mention that the export server won't do the cache task.
However, after I did the experiment, I found that the memory usage of
Ceph-client still went up to maximum value while writing the data
which means Ceph-client did the cache task.
So...are there other possible ways to solve my problem?
Thanks~~

Best Regards,
Sylar Shen

2011/4/12 Sage Weil <sage@newdream.net>:
> On Tue, 12 Apr 2011, Sylar Shen wrote:
>> Hi developers,
>> I know that the following question may not be relative to Ceph but
>> this confuses me a lot.
>> I am testing the function of re-exporting nfs by command "dd" without
>> using buffer.
>> That means I would like my data to be written to Ceph DFS directly
>> without passing by the buffer of client or nfs-client.
>> Here is my test architecture. Hope you could understand what I am
>> trying to express.....:p
>> ------------------------------------------------------------------------------
>>                 nfs                         ceph protocol
>> Client <------------>Ceph-Client <-------------------> Ceph DFS
>> ------------------------------------------------------------------------------
>> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
>> count=10000 oflag=dsync"
>> My purpose was to write the data from the client to Ceph DFS through
>> Ceph-client.
>> Ceph DFS was mounted by Ceph-client using Ceph protocol and
>> Ceph-client was mounted by client using nfs protocol.
>> I know that my dd command with argument oflag=dsync would write the
>> data to Ceph-client without using buffer on the client.
>> But I want my Ceph-client could also just pass the data to Ceph DFS
>> without using its buffer.
>> So.....is there any way I can run my scenario?
>> Or in other words, is there any method I can assure that Ceph-client
>> won't do the buffer task itself?
>
> I think what you're looking for is the 'sync' option in /etc/exports on
> the ceph-client machine.  See 'man 5 exports'.  That will slow things down
> considerably but will give you the consistency and safety you're looking
> for.
>
> sage
>



-- 
Best Regards,
Sylar Shen
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The cache problem of re-export NFS test

[Brian Chrisman]

I don't think 'sync' will prevent data from going into cache, it just
means that a copy has to be on stable storage/disk as well.  Whether
there's a copy in memory wont matter.

On Thu, Apr 14, 2011 at 1:52 AM, Sylar Shen <kimulaaa@gmail.com> wrote:
> Hi Sage,
> Thanks for your reply. I really appreciated.
> I tried what you suggested to set the argument "sync" in /etc/exports.
> I also checked "man 5 exports" and it showed that sync means reply to
> requests only after the changes have been committed to stable storage.
> It didn't mention that the export server won't do the cache task.
> However, after I did the experiment, I found that the memory usage of
> Ceph-client still went up to maximum value while writing the data
> which means Ceph-client did the cache task.
> So...are there other possible ways to solve my problem?
> Thanks~~
>
> Best Regards,
> Sylar Shen
>
> 2011/4/12 Sage Weil <sage@newdream.net>:
>> On Tue, 12 Apr 2011, Sylar Shen wrote:
>>> Hi developers,
>>> I know that the following question may not be relative to Ceph but
>>> this confuses me a lot.
>>> I am testing the function of re-exporting nfs by command "dd" without
>>> using buffer.
>>> That means I would like my data to be written to Ceph DFS directly
>>> without passing by the buffer of client or nfs-client.
>>> Here is my test architecture. Hope you could understand what I am
>>> trying to express.....:p
>>> ------------------------------------------------------------------------------
>>>                 nfs                         ceph protocol
>>> Client <------------>Ceph-Client <-------------------> Ceph DFS
>>> ------------------------------------------------------------------------------
>>> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
>>> count=10000 oflag=dsync"
>>> My purpose was to write the data from the client to Ceph DFS through
>>> Ceph-client.
>>> Ceph DFS was mounted by Ceph-client using Ceph protocol and
>>> Ceph-client was mounted by client using nfs protocol.
>>> I know that my dd command with argument oflag=dsync would write the
>>> data to Ceph-client without using buffer on the client.
>>> But I want my Ceph-client could also just pass the data to Ceph DFS
>>> without using its buffer.
>>> So.....is there any way I can run my scenario?
>>> Or in other words, is there any method I can assure that Ceph-client
>>> won't do the buffer task itself?
>>
>> I think what you're looking for is the 'sync' option in /etc/exports on
>> the ceph-client machine.  See 'man 5 exports'.  That will slow things down
>> considerably but will give you the consistency and safety you're looking
>> for.
>>
>> sage
>>
>
>
>
> --
> Best Regards,
> Sylar Shen
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The cache problem of re-export NFS test

[Sylar Shen]

Yes, after testing I found that "sync" will still do the cache in memory.
But what I want to know  is the real speed to write data into the disks.
I know it may not be safe without using memory and the speed may go very slow.
However, that's what I am trying to find out.
Thanks~~^^

2011/4/14 Brian Chrisman <brchrisman@gmail.com>:
> I don't think 'sync' will prevent data from going into cache, it just
> means that a copy has to be on stable storage/disk as well.  Whether
> there's a copy in memory wont matter.
>
> On Thu, Apr 14, 2011 at 1:52 AM, Sylar Shen <kimulaaa@gmail.com> wrote:
>> Hi Sage,
>> Thanks for your reply. I really appreciated.
>> I tried what you suggested to set the argument "sync" in /etc/exports.
>> I also checked "man 5 exports" and it showed that sync means reply to
>> requests only after the changes have been committed to stable storage.
>> It didn't mention that the export server won't do the cache task.
>> However, after I did the experiment, I found that the memory usage of
>> Ceph-client still went up to maximum value while writing the data
>> which means Ceph-client did the cache task.
>> So...are there other possible ways to solve my problem?
>> Thanks~~
>>
>> Best Regards,
>> Sylar Shen
>>
>> 2011/4/12 Sage Weil <sage@newdream.net>:
>>> On Tue, 12 Apr 2011, Sylar Shen wrote:
>>>> Hi developers,
>>>> I know that the following question may not be relative to Ceph but
>>>> this confuses me a lot.
>>>> I am testing the function of re-exporting nfs by command "dd" without
>>>> using buffer.
>>>> That means I would like my data to be written to Ceph DFS directly
>>>> without passing by the buffer of client or nfs-client.
>>>> Here is my test architecture. Hope you could understand what I am
>>>> trying to express.....:p
>>>> ------------------------------------------------------------------------------
>>>>                 nfs                         ceph protocol
>>>> Client <------------>Ceph-Client <-------------------> Ceph DFS
>>>> ------------------------------------------------------------------------------
>>>> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
>>>> count=10000 oflag=dsync"
>>>> My purpose was to write the data from the client to Ceph DFS through
>>>> Ceph-client.
>>>> Ceph DFS was mounted by Ceph-client using Ceph protocol and
>>>> Ceph-client was mounted by client using nfs protocol.
>>>> I know that my dd command with argument oflag=dsync would write the
>>>> data to Ceph-client without using buffer on the client.
>>>> But I want my Ceph-client could also just pass the data to Ceph DFS
>>>> without using its buffer.
>>>> So.....is there any way I can run my scenario?
>>>> Or in other words, is there any method I can assure that Ceph-client
>>>> won't do the buffer task itself?
>>>
>>> I think what you're looking for is the 'sync' option in /etc/exports on
>>> the ceph-client machine.  See 'man 5 exports'.  That will slow things down
>>> considerably but will give you the consistency and safety you're looking
>>> for.
>>>
>>> sage
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Sylar Shen
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>



-- 
Best Regards,
Sylar Shen
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The cache problem of re-export NFS test

[Henry Chang]

Have you tried the option no_wdelay with sync? Perhaps that's what you
want since it minimizes the cache effect.

2011/4/14 Sylar Shen <kimulaaa@gmail.com>:
> Yes, after testing I found that "sync" will still do the cache in memory.
> But what I want to know  is the real speed to write data into the disks.
> I know it may not be safe without using memory and the speed may go very slow.
> However, that's what I am trying to find out.
> Thanks~~^^
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The cache problem of re-export NFS test

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3523 bytes --]

On Thu, 14 Apr 2011, Sylar Shen wrote:
> Hi Sage,
> Thanks for your reply. I really appreciated.
> I tried what you suggested to set the argument "sync" in /etc/exports.
> I also checked "man 5 exports" and it showed that sync means reply to
> requests only after the changes have been committed to stable storage.
> It didn't mention that the export server won't do the cache task.
> However, after I did the experiment, I found that the memory usage of
> Ceph-client still went up to maximum value while writing the data
> which means Ceph-client did the cache task.

Right: 'sync' will make it write-through and flush the data to disk, but 
it will still keep a copy in the page cache.

> So...are there other possible ways to solve my problem?

Are you sure it's a problem?  Caching is usually a good thing.  If you're 
worried about safety, 'sync' solves that.  If you're worried about memory 
usage, you may not need to be.  The kernel will always use as much free 
ram as is available for caching fs data.  That's a normal and good thing.  
It will throw it out as soon as it needs the memory for something else.

I think the only case where that might be problematic is if you have other 
local file systems and want to make sure the memory is used for those 
instead of ceph.  I'm not aware of a way to do that...

sage




> Thanks~~
> 
> Best Regards,
> Sylar Shen
> 
> 2011/4/12 Sage Weil <sage@newdream.net>:
> > On Tue, 12 Apr 2011, Sylar Shen wrote:
> >> Hi developers,
> >> I know that the following question may not be relative to Ceph but
> >> this confuses me a lot.
> >> I am testing the function of re-exporting nfs by command "dd" without
> >> using buffer.
> >> That means I would like my data to be written to Ceph DFS directly
> >> without passing by the buffer of client or nfs-client.
> >> Here is my test architecture. Hope you could understand what I am
> >> trying to express.....:p
> >> ------------------------------------------------------------------------------
> >>                 nfs                         ceph protocol
> >> Client <------------>Ceph-Client <-------------------> Ceph DFS
> >> ------------------------------------------------------------------------------
> >> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
> >> count=10000 oflag=dsync"
> >> My purpose was to write the data from the client to Ceph DFS through
> >> Ceph-client.
> >> Ceph DFS was mounted by Ceph-client using Ceph protocol and
> >> Ceph-client was mounted by client using nfs protocol.
> >> I know that my dd command with argument oflag=dsync would write the
> >> data to Ceph-client without using buffer on the client.
> >> But I want my Ceph-client could also just pass the data to Ceph DFS
> >> without using its buffer.
> >> So.....is there any way I can run my scenario?
> >> Or in other words, is there any method I can assure that Ceph-client
> >> won't do the buffer task itself?
> >
> > I think what you're looking for is the 'sync' option in /etc/exports on
> > the ceph-client machine.  See 'man 5 exports'.  That will slow things down
> > considerably but will give you the consistency and safety you're looking
> > for.
> >
> > sage
> >
> 
> 
> 
> -- 
> Best Regards,
> Sylar Shen
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: The cache problem of re-export NFS test

[Colin McCabe]

On Thu, Apr 14, 2011 at 1:52 AM, Sylar Shen <kimulaaa@gmail.com> wrote:
> Hi Sage,
> Thanks for your reply. I really appreciated.
> I tried what you suggested to set the argument "sync" in /etc/exports.
> I also checked "man 5 exports" and it showed that sync means reply to
> requests only after the changes have been committed to stable storage.
> It didn't mention that the export server won't do the cache task.
> However, after I did the experiment, I found that the memory usage of
> Ceph-client still went up to maximum value while writing the data
> which means Ceph-client did the cache task.
> So...are there other possible ways to solve my problem?

I don't think the page cache is a "problem," really. It is there to
improve performance!

However, if you really want to remove pages from the page cache, there
is a little-known syscall called fadvise. fadvise(FADV_DONTNEED, ...)
will request that the kernel flush a region of the file out of the
page cache. I think someone wrote a patch for tar(1) that calls
fadvise(FADV_DONTNEED) the files after tar reads them, on the
assumption that they would probably not be read again in the near
future.

Also, there is this: http://code.google.com/p/pagecache-mangagement/

Of course, if you're trying to do performance measurement, I don't
think you want to tweak the page cache settings.

cheers,
Colin