All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFCv2][PATCH] add some drop_caches documentation and info messsge
@ 2010-09-16 16:50 ` Dave Hansen
  0 siblings, 0 replies; 10+ messages in thread
From: Dave Hansen @ 2010-09-16 16:50 UTC (permalink / raw)
  To: linux-kernel
  Cc: linux-mm, lnxninja, kamezawa.hiroyu, ebiederm, kosaki.motohiro,
	Dave Hansen


This version tones down the BUG_ON().  I also noticed that the
documentation fails to mention that more than just the inode
and dentry slabs are shrunk.

--

There is plenty of anecdotal evidence and a load of blog posts
suggesting that using "drop_caches" periodically keeps your system
running in "tip top shape".  Perhaps adding some kernel
documentation will increase the amount of accurate data on its use.

If we are not shrinking caches effectively, then we have real bugs.
Using drop_caches will simply mask the bugs and make them harder
to find, but certainly does not fix them, nor is it an appropriate
"workaround" to limit the size of the caches.

It's a great debugging tool, and is really handy for doing things
like repeatable benchmark runs.  So, add a bit more documentation
about it, and add a little KERN_NOTICE.  It should help developers
who are chasing down reclaim-related bugs.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/Documentation/sysctl/vm.txt |   33 ++++++++++++++++++++-----
 linux-2.6.git-dave/fs/drop_caches.c            |    2 +
 2 files changed, 29 insertions(+), 6 deletions(-)

diff -puN Documentation/sysctl/vm.txt~update-drop_caches-documentation Documentation/sysctl/vm.txt
--- linux-2.6.git/Documentation/sysctl/vm.txt~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
+++ linux-2.6.git-dave/Documentation/sysctl/vm.txt	2010-09-16 09:43:52.000000000 -0700
@@ -135,18 +135,39 @@ Setting this to zero disables periodic w
 
 drop_caches
 
-Writing to this will cause the kernel to drop clean caches, dentries and
-inodes from memory, causing that memory to become free.
+Writing to this will cause the kernel to drop clean caches, as well as
+reclaimable slab objects like dentries and inodes.  Once dropped, their
+memory becomes free.
 
 To free pagecache:
 	echo 1 > /proc/sys/vm/drop_caches
-To free dentries and inodes:
+To free reclaimable slab objects (includes dentries and inodes):
 	echo 2 > /proc/sys/vm/drop_caches
-To free pagecache, dentries and inodes:
+To free slab objects and pagecache:
 	echo 3 > /proc/sys/vm/drop_caches
 
-As this is a non-destructive operation and dirty objects are not freeable, the
-user should run `sync' first.
+This is a non-destructive operation and will not free any dirty objects.
+To increase the number of objects freed by this operation, the user may run
+`sync' prior to writing to /proc/sys/vm/drop_caches.  This will minimize the
+number of dirty objects on the system and create more candidates to be
+dropped.
+
+This file is not a means to control the growth of the various kernel caches
+(inodes, dentries, pagecache, etc...)  These objects are automatically
+reclaimed by the kernel when memory is needed elsewhere on the system.
+
+Use of this file can cause performance problems.  Since it discards cached
+objects, it may cost a significant amount of I/O and CPU to recreate the
+dropped objects, especially if they were under heavy use.  Because of this,
+use outside of a testing or debugging environment is not recommended.
+
+You may see informational messages in your kernel log when this file is
+used:
+
+	cat (1234): dropped kernel caches: 3
+
+These are informational only.  They do not mean that anything is wrong
+with your system.
 
 ==============================================================
 
diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
--- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
+++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
@@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
 {
 	proc_dointvec_minmax(table, write, buffer, length, ppos);
 	if (write) {
+		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
+			current->comm, task_pid_nr(current), sysctl_drop_caches);
 		if (sysctl_drop_caches & 1)
 			iterate_supers(drop_pagecache_sb, NULL);
 		if (sysctl_drop_caches & 2)
_

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [RFCv2][PATCH] add some drop_caches documentation and info messsge
@ 2010-09-16 16:50 ` Dave Hansen
  0 siblings, 0 replies; 10+ messages in thread
From: Dave Hansen @ 2010-09-16 16:50 UTC (permalink / raw)
  To: linux-kernel
  Cc: linux-mm, lnxninja, kamezawa.hiroyu, ebiederm, kosaki.motohiro,
	Dave Hansen


This version tones down the BUG_ON().  I also noticed that the
documentation fails to mention that more than just the inode
and dentry slabs are shrunk.

--

There is plenty of anecdotal evidence and a load of blog posts
suggesting that using "drop_caches" periodically keeps your system
running in "tip top shape".  Perhaps adding some kernel
documentation will increase the amount of accurate data on its use.

If we are not shrinking caches effectively, then we have real bugs.
Using drop_caches will simply mask the bugs and make them harder
to find, but certainly does not fix them, nor is it an appropriate
"workaround" to limit the size of the caches.

It's a great debugging tool, and is really handy for doing things
like repeatable benchmark runs.  So, add a bit more documentation
about it, and add a little KERN_NOTICE.  It should help developers
who are chasing down reclaim-related bugs.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/Documentation/sysctl/vm.txt |   33 ++++++++++++++++++++-----
 linux-2.6.git-dave/fs/drop_caches.c            |    2 +
 2 files changed, 29 insertions(+), 6 deletions(-)

diff -puN Documentation/sysctl/vm.txt~update-drop_caches-documentation Documentation/sysctl/vm.txt
--- linux-2.6.git/Documentation/sysctl/vm.txt~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
+++ linux-2.6.git-dave/Documentation/sysctl/vm.txt	2010-09-16 09:43:52.000000000 -0700
@@ -135,18 +135,39 @@ Setting this to zero disables periodic w
 
 drop_caches
 
-Writing to this will cause the kernel to drop clean caches, dentries and
-inodes from memory, causing that memory to become free.
+Writing to this will cause the kernel to drop clean caches, as well as
+reclaimable slab objects like dentries and inodes.  Once dropped, their
+memory becomes free.
 
 To free pagecache:
 	echo 1 > /proc/sys/vm/drop_caches
-To free dentries and inodes:
+To free reclaimable slab objects (includes dentries and inodes):
 	echo 2 > /proc/sys/vm/drop_caches
-To free pagecache, dentries and inodes:
+To free slab objects and pagecache:
 	echo 3 > /proc/sys/vm/drop_caches
 
-As this is a non-destructive operation and dirty objects are not freeable, the
-user should run `sync' first.
+This is a non-destructive operation and will not free any dirty objects.
+To increase the number of objects freed by this operation, the user may run
+`sync' prior to writing to /proc/sys/vm/drop_caches.  This will minimize the
+number of dirty objects on the system and create more candidates to be
+dropped.
+
+This file is not a means to control the growth of the various kernel caches
+(inodes, dentries, pagecache, etc...)  These objects are automatically
+reclaimed by the kernel when memory is needed elsewhere on the system.
+
+Use of this file can cause performance problems.  Since it discards cached
+objects, it may cost a significant amount of I/O and CPU to recreate the
+dropped objects, especially if they were under heavy use.  Because of this,
+use outside of a testing or debugging environment is not recommended.
+
+You may see informational messages in your kernel log when this file is
+used:
+
+	cat (1234): dropped kernel caches: 3
+
+These are informational only.  They do not mean that anything is wrong
+with your system.
 
 ==============================================================
 
diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
--- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
+++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
@@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
 {
 	proc_dointvec_minmax(table, write, buffer, length, ppos);
 	if (write) {
+		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
+			current->comm, task_pid_nr(current), sysctl_drop_caches);
 		if (sysctl_drop_caches & 1)
 			iterate_supers(drop_pagecache_sb, NULL);
 		if (sysctl_drop_caches & 2)
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
  2010-09-16 16:50 ` Dave Hansen
@ 2010-09-17  0:26   ` KOSAKI Motohiro
  -1 siblings, 0 replies; 10+ messages in thread
From: KOSAKI Motohiro @ 2010-09-17  0:26 UTC (permalink / raw)
  To: Dave Hansen
  Cc: kosaki.motohiro, linux-kernel, linux-mm, lnxninja,
	kamezawa.hiroyu, ebiederm

> diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
> --- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
> +++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
> @@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
>  {
>  	proc_dointvec_minmax(table, write, buffer, length, ppos);
>  	if (write) {
> +		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
> +			current->comm, task_pid_nr(current), sysctl_drop_caches);
>  		if (sysctl_drop_caches & 1)
>  			iterate_supers(drop_pagecache_sb, NULL);
>  		if (sysctl_drop_caches & 2)

Can't you print it only once?




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
@ 2010-09-17  0:26   ` KOSAKI Motohiro
  0 siblings, 0 replies; 10+ messages in thread
From: KOSAKI Motohiro @ 2010-09-17  0:26 UTC (permalink / raw)
  To: Dave Hansen
  Cc: kosaki.motohiro, linux-kernel, linux-mm, lnxninja,
	kamezawa.hiroyu, ebiederm

> diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
> --- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
> +++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
> @@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
>  {
>  	proc_dointvec_minmax(table, write, buffer, length, ppos);
>  	if (write) {
> +		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
> +			current->comm, task_pid_nr(current), sysctl_drop_caches);
>  		if (sysctl_drop_caches & 1)
>  			iterate_supers(drop_pagecache_sb, NULL);
>  		if (sysctl_drop_caches & 2)

Can't you print it only once?



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
  2010-09-17  0:26   ` KOSAKI Motohiro
@ 2010-09-17 15:54     ` Dave Hansen
  -1 siblings, 0 replies; 10+ messages in thread
From: Dave Hansen @ 2010-09-17 15:54 UTC (permalink / raw)
  To: KOSAKI Motohiro
  Cc: linux-kernel, linux-mm, lnxninja, kamezawa.hiroyu, ebiederm

On Fri, 2010-09-17 at 09:26 +0900, KOSAKI Motohiro wrote:
> > diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
> > --- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
> > +++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
> > @@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
> >  {
> >  	proc_dointvec_minmax(table, write, buffer, length, ppos);
> >  	if (write) {
> > +		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
> > +			current->comm, task_pid_nr(current), sysctl_drop_caches);
> >  		if (sysctl_drop_caches & 1)
> >  			iterate_supers(drop_pagecache_sb, NULL);
> >  		if (sysctl_drop_caches & 2)
> 
> Can't you print it only once?

Sure.  But, I also figured that somebody calling it every minute is
going to be much more interesting than something just on startup.
Should we printk_ratelimit() it, perhaps?

-- Dave


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
@ 2010-09-17 15:54     ` Dave Hansen
  0 siblings, 0 replies; 10+ messages in thread
From: Dave Hansen @ 2010-09-17 15:54 UTC (permalink / raw)
  To: KOSAKI Motohiro
  Cc: linux-kernel, linux-mm, lnxninja, kamezawa.hiroyu, ebiederm

On Fri, 2010-09-17 at 09:26 +0900, KOSAKI Motohiro wrote:
> > diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
> > --- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
> > +++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
> > @@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
> >  {
> >  	proc_dointvec_minmax(table, write, buffer, length, ppos);
> >  	if (write) {
> > +		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
> > +			current->comm, task_pid_nr(current), sysctl_drop_caches);
> >  		if (sysctl_drop_caches & 1)
> >  			iterate_supers(drop_pagecache_sb, NULL);
> >  		if (sysctl_drop_caches & 2)
> 
> Can't you print it only once?

Sure.  But, I also figured that somebody calling it every minute is
going to be much more interesting than something just on startup.
Should we printk_ratelimit() it, perhaps?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
  2010-09-17 15:54     ` Dave Hansen
@ 2010-09-21  0:57       ` KOSAKI Motohiro
  -1 siblings, 0 replies; 10+ messages in thread
From: KOSAKI Motohiro @ 2010-09-21  0:57 UTC (permalink / raw)
  To: Dave Hansen
  Cc: kosaki.motohiro, linux-kernel, linux-mm, lnxninja,
	kamezawa.hiroyu, ebiederm

> On Fri, 2010-09-17 at 09:26 +0900, KOSAKI Motohiro wrote:
> > > diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
> > > --- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
> > > +++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
> > > @@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
> > >  {
> > >  	proc_dointvec_minmax(table, write, buffer, length, ppos);
> > >  	if (write) {
> > > +		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
> > > +			current->comm, task_pid_nr(current), sysctl_drop_caches);
> > >  		if (sysctl_drop_caches & 1)
> > >  			iterate_supers(drop_pagecache_sb, NULL);
> > >  		if (sysctl_drop_caches & 2)
> > 
> > Can't you print it only once?
> 
> Sure.  But, I also figured that somebody calling it every minute is
> going to be much more interesting than something just on startup.
> Should we printk_ratelimit() it, perhaps?

Umm...

every minute drop_caches + printk_ratelimit() mean every drop_caches output
printk(). It seems annoying. I'm worry about that I'll see drop_caches's printk fill
my syslog.

But, It is not strong opinion. Because I don't use every minute drop_caches, then
I have no experience such usecase. It's up to you.




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
@ 2010-09-21  0:57       ` KOSAKI Motohiro
  0 siblings, 0 replies; 10+ messages in thread
From: KOSAKI Motohiro @ 2010-09-21  0:57 UTC (permalink / raw)
  To: Dave Hansen
  Cc: kosaki.motohiro, linux-kernel, linux-mm, lnxninja,
	kamezawa.hiroyu, ebiederm

> On Fri, 2010-09-17 at 09:26 +0900, KOSAKI Motohiro wrote:
> > > diff -puN fs/drop_caches.c~update-drop_caches-documentation fs/drop_caches.c
> > > --- linux-2.6.git/fs/drop_caches.c~update-drop_caches-documentation	2010-09-16 09:43:52.000000000 -0700
> > > +++ linux-2.6.git-dave/fs/drop_caches.c	2010-09-16 09:43:52.000000000 -0700
> > > @@ -47,6 +47,8 @@ int drop_caches_sysctl_handler(ctl_table
> > >  {
> > >  	proc_dointvec_minmax(table, write, buffer, length, ppos);
> > >  	if (write) {
> > > +		printk(KERN_NOTICE "%s (%d): dropped kernel caches: %d\n",
> > > +			current->comm, task_pid_nr(current), sysctl_drop_caches);
> > >  		if (sysctl_drop_caches & 1)
> > >  			iterate_supers(drop_pagecache_sb, NULL);
> > >  		if (sysctl_drop_caches & 2)
> > 
> > Can't you print it only once?
> 
> Sure.  But, I also figured that somebody calling it every minute is
> going to be much more interesting than something just on startup.
> Should we printk_ratelimit() it, perhaps?

Umm...

every minute drop_caches + printk_ratelimit() mean every drop_caches output
printk(). It seems annoying. I'm worry about that I'll see drop_caches's printk fill
my syslog.

But, It is not strong opinion. Because I don't use every minute drop_caches, then
I have no experience such usecase. It's up to you.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
  2010-09-16 16:50 ` Dave Hansen
@ 2010-09-21  1:00   ` KAMEZAWA Hiroyuki
  -1 siblings, 0 replies; 10+ messages in thread
From: KAMEZAWA Hiroyuki @ 2010-09-21  1:00 UTC (permalink / raw)
  To: Dave Hansen; +Cc: linux-kernel, linux-mm, lnxninja, ebiederm, kosaki.motohiro

On Thu, 16 Sep 2010 09:50:47 -0700
Dave Hansen <dave@linux.vnet.ibm.com> wrote:

> 
> This version tones down the BUG_ON().  I also noticed that the
> documentation fails to mention that more than just the inode
> and dentry slabs are shrunk.
> 
> --
> 
> There is plenty of anecdotal evidence and a load of blog posts
> suggesting that using "drop_caches" periodically keeps your system
> running in "tip top shape".  Perhaps adding some kernel
> documentation will increase the amount of accurate data on its use.
> 
> If we are not shrinking caches effectively, then we have real bugs.
> Using drop_caches will simply mask the bugs and make them harder
> to find, but certainly does not fix them, nor is it an appropriate
> "workaround" to limit the size of the caches.
> 
> It's a great debugging tool, and is really handy for doing things
> like repeatable benchmark runs.  So, add a bit more documentation
> about it, and add a little KERN_NOTICE.  It should help developers
> who are chasing down reclaim-related bugs.
> 
> Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>

Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

thanks.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFCv2][PATCH] add some drop_caches documentation and info messsge
@ 2010-09-21  1:00   ` KAMEZAWA Hiroyuki
  0 siblings, 0 replies; 10+ messages in thread
From: KAMEZAWA Hiroyuki @ 2010-09-21  1:00 UTC (permalink / raw)
  To: Dave Hansen; +Cc: linux-kernel, linux-mm, lnxninja, ebiederm, kosaki.motohiro

On Thu, 16 Sep 2010 09:50:47 -0700
Dave Hansen <dave@linux.vnet.ibm.com> wrote:

> 
> This version tones down the BUG_ON().  I also noticed that the
> documentation fails to mention that more than just the inode
> and dentry slabs are shrunk.
> 
> --
> 
> There is plenty of anecdotal evidence and a load of blog posts
> suggesting that using "drop_caches" periodically keeps your system
> running in "tip top shape".  Perhaps adding some kernel
> documentation will increase the amount of accurate data on its use.
> 
> If we are not shrinking caches effectively, then we have real bugs.
> Using drop_caches will simply mask the bugs and make them harder
> to find, but certainly does not fix them, nor is it an appropriate
> "workaround" to limit the size of the caches.
> 
> It's a great debugging tool, and is really handy for doing things
> like repeatable benchmark runs.  So, add a bit more documentation
> about it, and add a little KERN_NOTICE.  It should help developers
> who are chasing down reclaim-related bugs.
> 
> Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>

Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2010-09-21  1:05 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-09-16 16:50 [RFCv2][PATCH] add some drop_caches documentation and info messsge Dave Hansen
2010-09-16 16:50 ` Dave Hansen
2010-09-17  0:26 ` KOSAKI Motohiro
2010-09-17  0:26   ` KOSAKI Motohiro
2010-09-17 15:54   ` Dave Hansen
2010-09-17 15:54     ` Dave Hansen
2010-09-21  0:57     ` KOSAKI Motohiro
2010-09-21  0:57       ` KOSAKI Motohiro
2010-09-21  1:00 ` KAMEZAWA Hiroyuki
2010-09-21  1:00   ` KAMEZAWA Hiroyuki

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.