* Re: The cache problem of re-export NFS test
2011-04-14 8:52 ` Sylar Shen
@ 2011-04-14 12:30 ` Brian Chrisman
2011-04-14 12:49 ` Sylar Shen
2011-04-14 17:12 ` Sage Weil
2011-04-14 17:56 ` Colin McCabe
2 siblings, 1 reply; 8+ messages in thread
From: Brian Chrisman @ 2011-04-14 12:30 UTC (permalink / raw)
To: Sylar Shen; +Cc: Sage Weil, ceph-devel
I don't think 'sync' will prevent data from going into cache, it just
means that a copy has to be on stable storage/disk as well. Whether
there's a copy in memory wont matter.
On Thu, Apr 14, 2011 at 1:52 AM, Sylar Shen <kimulaaa@gmail.com> wrote:
> Hi Sage,
> Thanks for your reply. I really appreciated.
> I tried what you suggested to set the argument "sync" in /etc/exports.
> I also checked "man 5 exports" and it showed that sync means reply to
> requests only after the changes have been committed to stable storage.
> It didn't mention that the export server won't do the cache task.
> However, after I did the experiment, I found that the memory usage of
> Ceph-client still went up to maximum value while writing the data
> which means Ceph-client did the cache task.
> So...are there other possible ways to solve my problem?
> Thanks~~
>
> Best Regards,
> Sylar Shen
>
> 2011/4/12 Sage Weil <sage@newdream.net>:
>> On Tue, 12 Apr 2011, Sylar Shen wrote:
>>> Hi developers,
>>> I know that the following question may not be relative to Ceph but
>>> this confuses me a lot.
>>> I am testing the function of re-exporting nfs by command "dd" without
>>> using buffer.
>>> That means I would like my data to be written to Ceph DFS directly
>>> without passing by the buffer of client or nfs-client.
>>> Here is my test architecture. Hope you could understand what I am
>>> trying to express.....:p
>>> ------------------------------------------------------------------------------
>>> nfs ceph protocol
>>> Client <------------>Ceph-Client <-------------------> Ceph DFS
>>> ------------------------------------------------------------------------------
>>> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
>>> count=10000 oflag=dsync"
>>> My purpose was to write the data from the client to Ceph DFS through
>>> Ceph-client.
>>> Ceph DFS was mounted by Ceph-client using Ceph protocol and
>>> Ceph-client was mounted by client using nfs protocol.
>>> I know that my dd command with argument oflag=dsync would write the
>>> data to Ceph-client without using buffer on the client.
>>> But I want my Ceph-client could also just pass the data to Ceph DFS
>>> without using its buffer.
>>> So.....is there any way I can run my scenario?
>>> Or in other words, is there any method I can assure that Ceph-client
>>> won't do the buffer task itself?
>>
>> I think what you're looking for is the 'sync' option in /etc/exports on
>> the ceph-client machine. See 'man 5 exports'. That will slow things down
>> considerably but will give you the consistency and safety you're looking
>> for.
>>
>> sage
>>
>
>
>
> --
> Best Regards,
> Sylar Shen
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The cache problem of re-export NFS test
2011-04-14 12:30 ` Brian Chrisman
@ 2011-04-14 12:49 ` Sylar Shen
2011-04-14 15:19 ` Henry Chang
0 siblings, 1 reply; 8+ messages in thread
From: Sylar Shen @ 2011-04-14 12:49 UTC (permalink / raw)
To: Brian Chrisman; +Cc: Sage Weil, ceph-devel
Yes, after testing I found that "sync" will still do the cache in memory.
But what I want to know is the real speed to write data into the disks.
I know it may not be safe without using memory and the speed may go very slow.
However, that's what I am trying to find out.
Thanks~~^^
2011/4/14 Brian Chrisman <brchrisman@gmail.com>:
> I don't think 'sync' will prevent data from going into cache, it just
> means that a copy has to be on stable storage/disk as well. Whether
> there's a copy in memory wont matter.
>
> On Thu, Apr 14, 2011 at 1:52 AM, Sylar Shen <kimulaaa@gmail.com> wrote:
>> Hi Sage,
>> Thanks for your reply. I really appreciated.
>> I tried what you suggested to set the argument "sync" in /etc/exports.
>> I also checked "man 5 exports" and it showed that sync means reply to
>> requests only after the changes have been committed to stable storage.
>> It didn't mention that the export server won't do the cache task.
>> However, after I did the experiment, I found that the memory usage of
>> Ceph-client still went up to maximum value while writing the data
>> which means Ceph-client did the cache task.
>> So...are there other possible ways to solve my problem?
>> Thanks~~
>>
>> Best Regards,
>> Sylar Shen
>>
>> 2011/4/12 Sage Weil <sage@newdream.net>:
>>> On Tue, 12 Apr 2011, Sylar Shen wrote:
>>>> Hi developers,
>>>> I know that the following question may not be relative to Ceph but
>>>> this confuses me a lot.
>>>> I am testing the function of re-exporting nfs by command "dd" without
>>>> using buffer.
>>>> That means I would like my data to be written to Ceph DFS directly
>>>> without passing by the buffer of client or nfs-client.
>>>> Here is my test architecture. Hope you could understand what I am
>>>> trying to express.....:p
>>>> ------------------------------------------------------------------------------
>>>> nfs ceph protocol
>>>> Client <------------>Ceph-Client <-------------------> Ceph DFS
>>>> ------------------------------------------------------------------------------
>>>> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
>>>> count=10000 oflag=dsync"
>>>> My purpose was to write the data from the client to Ceph DFS through
>>>> Ceph-client.
>>>> Ceph DFS was mounted by Ceph-client using Ceph protocol and
>>>> Ceph-client was mounted by client using nfs protocol.
>>>> I know that my dd command with argument oflag=dsync would write the
>>>> data to Ceph-client without using buffer on the client.
>>>> But I want my Ceph-client could also just pass the data to Ceph DFS
>>>> without using its buffer.
>>>> So.....is there any way I can run my scenario?
>>>> Or in other words, is there any method I can assure that Ceph-client
>>>> won't do the buffer task itself?
>>>
>>> I think what you're looking for is the 'sync' option in /etc/exports on
>>> the ceph-client machine. See 'man 5 exports'. That will slow things down
>>> considerably but will give you the consistency and safety you're looking
>>> for.
>>>
>>> sage
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Sylar Shen
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
--
Best Regards,
Sylar Shen
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The cache problem of re-export NFS test
2011-04-14 12:49 ` Sylar Shen
@ 2011-04-14 15:19 ` Henry Chang
0 siblings, 0 replies; 8+ messages in thread
From: Henry Chang @ 2011-04-14 15:19 UTC (permalink / raw)
To: Sylar Shen; +Cc: Brian Chrisman, Sage Weil, ceph-devel
Have you tried the option no_wdelay with sync? Perhaps that's what you
want since it minimizes the cache effect.
2011/4/14 Sylar Shen <kimulaaa@gmail.com>:
> Yes, after testing I found that "sync" will still do the cache in memory.
> But what I want to know is the real speed to write data into the disks.
> I know it may not be safe without using memory and the speed may go very slow.
> However, that's what I am trying to find out.
> Thanks~~^^
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The cache problem of re-export NFS test
2011-04-14 8:52 ` Sylar Shen
2011-04-14 12:30 ` Brian Chrisman
@ 2011-04-14 17:12 ` Sage Weil
2011-04-14 17:56 ` Colin McCabe
2 siblings, 0 replies; 8+ messages in thread
From: Sage Weil @ 2011-04-14 17:12 UTC (permalink / raw)
To: Sylar Shen; +Cc: ceph-devel
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3523 bytes --]
On Thu, 14 Apr 2011, Sylar Shen wrote:
> Hi Sage,
> Thanks for your reply. I really appreciated.
> I tried what you suggested to set the argument "sync" in /etc/exports.
> I also checked "man 5 exports" and it showed that sync means reply to
> requests only after the changes have been committed to stable storage.
> It didn't mention that the export server won't do the cache task.
> However, after I did the experiment, I found that the memory usage of
> Ceph-client still went up to maximum value while writing the data
> which means Ceph-client did the cache task.
Right: 'sync' will make it write-through and flush the data to disk, but
it will still keep a copy in the page cache.
> So...are there other possible ways to solve my problem?
Are you sure it's a problem? Caching is usually a good thing. If you're
worried about safety, 'sync' solves that. If you're worried about memory
usage, you may not need to be. The kernel will always use as much free
ram as is available for caching fs data. That's a normal and good thing.
It will throw it out as soon as it needs the memory for something else.
I think the only case where that might be problematic is if you have other
local file systems and want to make sure the memory is used for those
instead of ceph. I'm not aware of a way to do that...
sage
> Thanks~~
>
> Best Regards,
> Sylar Shen
>
> 2011/4/12 Sage Weil <sage@newdream.net>:
> > On Tue, 12 Apr 2011, Sylar Shen wrote:
> >> Hi developers,
> >> I know that the following question may not be relative to Ceph but
> >> this confuses me a lot.
> >> I am testing the function of re-exporting nfs by command "dd" without
> >> using buffer.
> >> That means I would like my data to be written to Ceph DFS directly
> >> without passing by the buffer of client or nfs-client.
> >> Here is my test architecture. Hope you could understand what I am
> >> trying to express.....:p
> >> ------------------------------------------------------------------------------
> >> nfs ceph protocol
> >> Client <------------>Ceph-Client <-------------------> Ceph DFS
> >> ------------------------------------------------------------------------------
> >> I used the command "dd if=/dev/zero of=/mnt/ceph/test bs=64k
> >> count=10000 oflag=dsync"
> >> My purpose was to write the data from the client to Ceph DFS through
> >> Ceph-client.
> >> Ceph DFS was mounted by Ceph-client using Ceph protocol and
> >> Ceph-client was mounted by client using nfs protocol.
> >> I know that my dd command with argument oflag=dsync would write the
> >> data to Ceph-client without using buffer on the client.
> >> But I want my Ceph-client could also just pass the data to Ceph DFS
> >> without using its buffer.
> >> So.....is there any way I can run my scenario?
> >> Or in other words, is there any method I can assure that Ceph-client
> >> won't do the buffer task itself?
> >
> > I think what you're looking for is the 'sync' option in /etc/exports on
> > the ceph-client machine. See 'man 5 exports'. That will slow things down
> > considerably but will give you the consistency and safety you're looking
> > for.
> >
> > sage
> >
>
>
>
> --
> Best Regards,
> Sylar Shen
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The cache problem of re-export NFS test
2011-04-14 8:52 ` Sylar Shen
2011-04-14 12:30 ` Brian Chrisman
2011-04-14 17:12 ` Sage Weil
@ 2011-04-14 17:56 ` Colin McCabe
2 siblings, 0 replies; 8+ messages in thread
From: Colin McCabe @ 2011-04-14 17:56 UTC (permalink / raw)
To: Sylar Shen; +Cc: Sage Weil, ceph-devel
On Thu, Apr 14, 2011 at 1:52 AM, Sylar Shen <kimulaaa@gmail.com> wrote:
> Hi Sage,
> Thanks for your reply. I really appreciated.
> I tried what you suggested to set the argument "sync" in /etc/exports.
> I also checked "man 5 exports" and it showed that sync means reply to
> requests only after the changes have been committed to stable storage.
> It didn't mention that the export server won't do the cache task.
> However, after I did the experiment, I found that the memory usage of
> Ceph-client still went up to maximum value while writing the data
> which means Ceph-client did the cache task.
> So...are there other possible ways to solve my problem?
I don't think the page cache is a "problem," really. It is there to
improve performance!
However, if you really want to remove pages from the page cache, there
is a little-known syscall called fadvise. fadvise(FADV_DONTNEED, ...)
will request that the kernel flush a region of the file out of the
page cache. I think someone wrote a patch for tar(1) that calls
fadvise(FADV_DONTNEED) the files after tar reads them, on the
assumption that they would probably not be read again in the near
future.
Also, there is this: http://code.google.com/p/pagecache-mangagement/
Of course, if you're trying to do performance measurement, I don't
think you want to tweak the page cache settings.
cheers,
Colin
^ permalink raw reply [flat|nested] 8+ messages in thread