* Huge lookup when recursively mkdir
@ 2017-10-19 16:49 Xiaoxi Chen
2017-10-20 2:55 ` Yan, Zheng
0 siblings, 1 reply; 11+ messages in thread
From: Xiaoxi Chen @ 2017-10-19 16:49 UTC (permalink / raw)
To: Ceph Development, John Spray, Yan, Zheng
Hi,
I am seeing a lot of lookup request when doing recursive mkdir.
The workload behavior is like:
mkdir DIR0
mkdir DIR0/DIR1
mkdir DIR0/DIR1/DIR2
....
mkdir DIR0/DIR1/DIR2......./DIR7
create DIR0/DIR1/DIR2......./DIR7/FILE1
and concurrently run on 50+ clients, the dir name in different
clients may or maynot be the same.
from the admin socket I was seeing ~50K create requests, but
got 400K lookup requests. The lookup eat up most of the mds capability
so file create is slow.
Where is the lookup comes from and can we have anyway to
optimize it out ?
Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-19 16:49 Huge lookup when recursively mkdir Xiaoxi Chen
@ 2017-10-20 2:55 ` Yan, Zheng
2017-10-20 7:28 ` Xiaoxi Chen
0 siblings, 1 reply; 11+ messages in thread
From: Yan, Zheng @ 2017-10-20 2:55 UTC (permalink / raw)
To: Xiaoxi Chen; +Cc: Ceph Development, John Spray
On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
> Hi,
>
> I am seeing a lot of lookup request when doing recursive mkdir.
> The workload behavior is like:
> mkdir DIR0
> mkdir DIR0/DIR1
> mkdir DIR0/DIR1/DIR2
> ....
> mkdir DIR0/DIR1/DIR2......./DIR7
> create DIR0/DIR1/DIR2......./DIR7/FILE1
>
> and concurrently run on 50+ clients, the dir name in different
> clients may or maynot be the same.
>
> from the admin socket I was seeing ~50K create requests, but
> got 400K lookup requests. The lookup eat up most of the mds capability
> so file create is slow.
>
> Where is the lookup comes from and can we have anyway to
> optimize it out ?
>
I don't see this behavior when running following commands in 4.13
kernel client and luminous version ceph-fuse. which client do you use?
mkdir d1
mkdir d1/d2
mkdir d1/d2/d3
mkdir d1/d2/d3/d4/
mkdir d1/d2/d3/d4/d5
touch d1/d2/d3/d4/d5/f
> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-20 2:55 ` Yan, Zheng
@ 2017-10-20 7:28 ` Xiaoxi Chen
2017-10-20 8:39 ` Yan, Zheng
0 siblings, 1 reply; 11+ messages in thread
From: Xiaoxi Chen @ 2017-10-20 7:28 UTC (permalink / raw)
To: Yan, Zheng; +Cc: Ceph Development, John Spray
Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
I extract the logical of file creation in our workload into a
reproducer , like below.
Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
I thought the lookup is to open the directory tree so I tried to
pre-make most of the dirs , use ls -i trying to read the dentries and
cache it, then re-run the reproducer, seems nothing different..
#include <sys/stat.h>
#include <fcntl.h>
int create_file(char * base, int count, int max, int depth)
{
int i;
for(i=0; i<count; i++) {
char dir[256];
int mydir = rand() % max;
sprintf(dir, "%s/%d", path, mydir);
if (depth >=1) {
mkdir(dir,0777);
create_dir(dir, count, max, depth - 1);
} else {
int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
printf("opened path : %s = %d\n", path, fd);
close(fd);
}
}
}
int main(int argc, char argv[])
{
char path[256];
while(1) {
create_file("/import/SQL01", 1, 4 ,10);
}
}
2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>> Hi,
>>
>> I am seeing a lot of lookup request when doing recursive mkdir.
>> The workload behavior is like:
>> mkdir DIR0
>> mkdir DIR0/DIR1
>> mkdir DIR0/DIR1/DIR2
>> ....
>> mkdir DIR0/DIR1/DIR2......./DIR7
>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>
>> and concurrently run on 50+ clients, the dir name in different
>> clients may or maynot be the same.
>>
>> from the admin socket I was seeing ~50K create requests, but
>> got 400K lookup requests. The lookup eat up most of the mds capability
>> so file create is slow.
>>
>> Where is the lookup comes from and can we have anyway to
>> optimize it out ?
>>
>
> I don't see this behavior when running following commands in 4.13
> kernel client and luminous version ceph-fuse. which client do you use?
>
> mkdir d1
> mkdir d1/d2
> mkdir d1/d2/d3
> mkdir d1/d2/d3/d4/
> mkdir d1/d2/d3/d4/d5
> touch d1/d2/d3/d4/d5/f
>
>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-20 7:28 ` Xiaoxi Chen
@ 2017-10-20 8:39 ` Yan, Zheng
2017-10-20 16:54 ` Xiaoxi Chen
0 siblings, 1 reply; 11+ messages in thread
From: Yan, Zheng @ 2017-10-20 8:39 UTC (permalink / raw)
To: Xiaoxi Chen; +Cc: Ceph Development, John Spray
On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>
> I extract the logical of file creation in our workload into a
> reproducer , like below.
>
> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
> I thought the lookup is to open the directory tree so I tried to
> pre-make most of the dirs , use ls -i trying to read the dentries and
> cache it, then re-run the reproducer, seems nothing different..
>
> #include <sys/stat.h>
> #include <fcntl.h>
> int create_file(char * base, int count, int max, int depth)
> {
> int i;
> for(i=0; i<count; i++) {
> char dir[256];
> int mydir = rand() % max;
> sprintf(dir, "%s/%d", path, mydir);
> if (depth >=1) {
> mkdir(dir,0777);
> create_dir(dir, count, max, depth - 1);
> } else {
> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
> printf("opened path : %s = %d\n", path, fd);
> close(fd);
> }
> }
> }
> int main(int argc, char argv[])
> {
> char path[256];
> while(1) {
> create_file("/import/SQL01", 1, 4 ,10);
> }
> }
>
still don't see this behavior on 4.13 kernel. I suspect there is
something wrong with dentry lease. please check if your kernel
include:
commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
The first commit can cause this issue, the second one fixes it.
Regards
Yan, Zheng
>
>
> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>> Hi,
>>>
>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>> The workload behavior is like:
>>> mkdir DIR0
>>> mkdir DIR0/DIR1
>>> mkdir DIR0/DIR1/DIR2
>>> ....
>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>
>>> and concurrently run on 50+ clients, the dir name in different
>>> clients may or maynot be the same.
>>>
>>> from the admin socket I was seeing ~50K create requests, but
>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>> so file create is slow.
>>>
>>> Where is the lookup comes from and can we have anyway to
>>> optimize it out ?
>>>
>>
>> I don't see this behavior when running following commands in 4.13
>> kernel client and luminous version ceph-fuse. which client do you use?
>>
>> mkdir d1
>> mkdir d1/d2
>> mkdir d1/d2/d3
>> mkdir d1/d2/d3/d4/
>> mkdir d1/d2/d3/d4/d5
>> touch d1/d2/d3/d4/d5/f
>>
>>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-20 8:39 ` Yan, Zheng
@ 2017-10-20 16:54 ` Xiaoxi Chen
2017-10-20 18:09 ` Xiaoxi Chen
0 siblings, 1 reply; 11+ messages in thread
From: Xiaoxi Chen @ 2017-10-20 16:54 UTC (permalink / raw)
To: Yan, Zheng; +Cc: Ceph Development, John Spray
Thanks, will check.
A general question, does cephfs kernel client drop dentries/inode
cache aggressively? What I know is if MDS issue
CEPH_SESSION_RECALL_STATE, client will drop, but is there other cases
client will drop cache?
2017-10-20 16:39 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
> On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>>
>> I extract the logical of file creation in our workload into a
>> reproducer , like below.
>>
>> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
>> I thought the lookup is to open the directory tree so I tried to
>> pre-make most of the dirs , use ls -i trying to read the dentries and
>> cache it, then re-run the reproducer, seems nothing different..
>>
>> #include <sys/stat.h>
>> #include <fcntl.h>
>> int create_file(char * base, int count, int max, int depth)
>> {
>> int i;
>> for(i=0; i<count; i++) {
>> char dir[256];
>> int mydir = rand() % max;
>> sprintf(dir, "%s/%d", path, mydir);
>> if (depth >=1) {
>> mkdir(dir,0777);
>> create_dir(dir, count, max, depth - 1);
>> } else {
>> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
>> printf("opened path : %s = %d\n", path, fd);
>> close(fd);
>> }
>> }
>> }
>> int main(int argc, char argv[])
>> {
>> char path[256];
>> while(1) {
>> create_file("/import/SQL01", 1, 4 ,10);
>> }
>> }
>>
>
> still don't see this behavior on 4.13 kernel. I suspect there is
> something wrong with dentry lease. please check if your kernel
> include:
>
> commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
> commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
>
> The first commit can cause this issue, the second one fixes it.
>
> Regards
> Yan, Zheng
>
>>
>>
>> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>>> The workload behavior is like:
>>>> mkdir DIR0
>>>> mkdir DIR0/DIR1
>>>> mkdir DIR0/DIR1/DIR2
>>>> ....
>>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>>
>>>> and concurrently run on 50+ clients, the dir name in different
>>>> clients may or maynot be the same.
>>>>
>>>> from the admin socket I was seeing ~50K create requests, but
>>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>>> so file create is slow.
>>>>
>>>> Where is the lookup comes from and can we have anyway to
>>>> optimize it out ?
>>>>
>>>
>>> I don't see this behavior when running following commands in 4.13
>>> kernel client and luminous version ceph-fuse. which client do you use?
>>>
>>> mkdir d1
>>> mkdir d1/d2
>>> mkdir d1/d2/d3
>>> mkdir d1/d2/d3/d4/
>>> mkdir d1/d2/d3/d4/d5
>>> touch d1/d2/d3/d4/d5/f
>>>
>>>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-20 16:54 ` Xiaoxi Chen
@ 2017-10-20 18:09 ` Xiaoxi Chen
2017-10-22 15:27 ` Xiaoxi Chen
0 siblings, 1 reply; 11+ messages in thread
From: Xiaoxi Chen @ 2017-10-20 18:09 UTC (permalink / raw)
To: Yan, Zheng; +Cc: Ceph Development, John Spray
@Zheng, my kernel doesn't even has c3f4688a08f. But 200fd27 ("ceph:
use lookup request to revalidate dentry") is there.
2017-10-21 0:54 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
> Thanks, will check.
>
> A general question, does cephfs kernel client drop dentries/inode
> cache aggressively? What I know is if MDS issue
> CEPH_SESSION_RECALL_STATE, client will drop, but is there other cases
> client will drop cache?
>
>
>
> 2017-10-20 16:39 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>> On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>>>
>>> I extract the logical of file creation in our workload into a
>>> reproducer , like below.
>>>
>>> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
>>> I thought the lookup is to open the directory tree so I tried to
>>> pre-make most of the dirs , use ls -i trying to read the dentries and
>>> cache it, then re-run the reproducer, seems nothing different..
>>>
>>> #include <sys/stat.h>
>>> #include <fcntl.h>
>>> int create_file(char * base, int count, int max, int depth)
>>> {
>>> int i;
>>> for(i=0; i<count; i++) {
>>> char dir[256];
>>> int mydir = rand() % max;
>>> sprintf(dir, "%s/%d", path, mydir);
>>> if (depth >=1) {
>>> mkdir(dir,0777);
>>> create_dir(dir, count, max, depth - 1);
>>> } else {
>>> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
>>> printf("opened path : %s = %d\n", path, fd);
>>> close(fd);
>>> }
>>> }
>>> }
>>> int main(int argc, char argv[])
>>> {
>>> char path[256];
>>> while(1) {
>>> create_file("/import/SQL01", 1, 4 ,10);
>>> }
>>> }
>>>
>>
>> still don't see this behavior on 4.13 kernel. I suspect there is
>> something wrong with dentry lease. please check if your kernel
>> include:
>>
>> commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
>> commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
>>
>> The first commit can cause this issue, the second one fixes it.
>>
>> Regards
>> Yan, Zheng
>>
>>>
>>>
>>> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>>>> The workload behavior is like:
>>>>> mkdir DIR0
>>>>> mkdir DIR0/DIR1
>>>>> mkdir DIR0/DIR1/DIR2
>>>>> ....
>>>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>>>
>>>>> and concurrently run on 50+ clients, the dir name in different
>>>>> clients may or maynot be the same.
>>>>>
>>>>> from the admin socket I was seeing ~50K create requests, but
>>>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>>>> so file create is slow.
>>>>>
>>>>> Where is the lookup comes from and can we have anyway to
>>>>> optimize it out ?
>>>>>
>>>>
>>>> I don't see this behavior when running following commands in 4.13
>>>> kernel client and luminous version ceph-fuse. which client do you use?
>>>>
>>>> mkdir d1
>>>> mkdir d1/d2
>>>> mkdir d1/d2/d3
>>>> mkdir d1/d2/d3/d4/
>>>> mkdir d1/d2/d3/d4/d5
>>>> touch d1/d2/d3/d4/d5/f
>>>>
>>>>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-20 18:09 ` Xiaoxi Chen
@ 2017-10-22 15:27 ` Xiaoxi Chen
2017-10-23 0:44 ` Yan, Zheng
0 siblings, 1 reply; 11+ messages in thread
From: Xiaoxi Chen @ 2017-10-22 15:27 UTC (permalink / raw)
To: Yan, Zheng; +Cc: Ceph Development, John Spray
To add another data point, switched to ceph-fuse 12.2.0, still seeing
lots of lookup.
lookup avg 1892
mkdir avg 367
create avg 222
open avg 228
2017-10-21 2:09 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
> @Zheng, my kernel doesn't even has c3f4688a08f. But 200fd27 ("ceph:
> use lookup request to revalidate dentry") is there.
>
> 2017-10-21 0:54 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>> Thanks, will check.
>>
>> A general question, does cephfs kernel client drop dentries/inode
>> cache aggressively? What I know is if MDS issue
>> CEPH_SESSION_RECALL_STATE, client will drop, but is there other cases
>> client will drop cache?
>>
>>
>>
>> 2017-10-20 16:39 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>> On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>>>>
>>>> I extract the logical of file creation in our workload into a
>>>> reproducer , like below.
>>>>
>>>> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
>>>> I thought the lookup is to open the directory tree so I tried to
>>>> pre-make most of the dirs , use ls -i trying to read the dentries and
>>>> cache it, then re-run the reproducer, seems nothing different..
>>>>
>>>> #include <sys/stat.h>
>>>> #include <fcntl.h>
>>>> int create_file(char * base, int count, int max, int depth)
>>>> {
>>>> int i;
>>>> for(i=0; i<count; i++) {
>>>> char dir[256];
>>>> int mydir = rand() % max;
>>>> sprintf(dir, "%s/%d", path, mydir);
>>>> if (depth >=1) {
>>>> mkdir(dir,0777);
>>>> create_dir(dir, count, max, depth - 1);
>>>> } else {
>>>> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
>>>> printf("opened path : %s = %d\n", path, fd);
>>>> close(fd);
>>>> }
>>>> }
>>>> }
>>>> int main(int argc, char argv[])
>>>> {
>>>> char path[256];
>>>> while(1) {
>>>> create_file("/import/SQL01", 1, 4 ,10);
>>>> }
>>>> }
>>>>
>>>
>>> still don't see this behavior on 4.13 kernel. I suspect there is
>>> something wrong with dentry lease. please check if your kernel
>>> include:
>>>
>>> commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
>>> commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
>>>
>>> The first commit can cause this issue, the second one fixes it.
>>>
>>> Regards
>>> Yan, Zheng
>>>
>>>>
>>>>
>>>> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>>>>> The workload behavior is like:
>>>>>> mkdir DIR0
>>>>>> mkdir DIR0/DIR1
>>>>>> mkdir DIR0/DIR1/DIR2
>>>>>> ....
>>>>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>>>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>>>>
>>>>>> and concurrently run on 50+ clients, the dir name in different
>>>>>> clients may or maynot be the same.
>>>>>>
>>>>>> from the admin socket I was seeing ~50K create requests, but
>>>>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>>>>> so file create is slow.
>>>>>>
>>>>>> Where is the lookup comes from and can we have anyway to
>>>>>> optimize it out ?
>>>>>>
>>>>>
>>>>> I don't see this behavior when running following commands in 4.13
>>>>> kernel client and luminous version ceph-fuse. which client do you use?
>>>>>
>>>>> mkdir d1
>>>>> mkdir d1/d2
>>>>> mkdir d1/d2/d3
>>>>> mkdir d1/d2/d3/d4/
>>>>> mkdir d1/d2/d3/d4/d5
>>>>> touch d1/d2/d3/d4/d5/f
>>>>>
>>>>>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-22 15:27 ` Xiaoxi Chen
@ 2017-10-23 0:44 ` Yan, Zheng
2017-10-23 6:01 ` Xiaoxi Chen
0 siblings, 1 reply; 11+ messages in thread
From: Yan, Zheng @ 2017-10-23 0:44 UTC (permalink / raw)
To: Xiaoxi Chen; +Cc: Ceph Development, John Spray
On Sun, Oct 22, 2017 at 11:27 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
> To add another data point, switched to ceph-fuse 12.2.0, still seeing
> lots of lookup.
> lookup avg 1892
> mkdir avg 367
> create avg 222
> open avg 228
>
But in your test, mkdir avg was about 1.5 times of open avg. I think
your test created millions of directories, lookups were from cache
miss. You can try enlarging client_cache_size. But I don't think it
will help much when active set of directory are so large.
> 2017-10-21 2:09 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>> @Zheng, my kernel doesn't even has c3f4688a08f. But 200fd27 ("ceph:
>> use lookup request to revalidate dentry") is there.
>>
>> 2017-10-21 0:54 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>>> Thanks, will check.
>>>
>>> A general question, does cephfs kernel client drop dentries/inode
>>> cache aggressively? What I know is if MDS issue
>>> CEPH_SESSION_RECALL_STATE, client will drop, but is there other cases
>>> client will drop cache?
>>>
>>>
>>>
>>> 2017-10-20 16:39 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>> On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>>>>>
>>>>> I extract the logical of file creation in our workload into a
>>>>> reproducer , like below.
>>>>>
>>>>> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
>>>>> I thought the lookup is to open the directory tree so I tried to
>>>>> pre-make most of the dirs , use ls -i trying to read the dentries and
>>>>> cache it, then re-run the reproducer, seems nothing different..
>>>>>
>>>>> #include <sys/stat.h>
>>>>> #include <fcntl.h>
>>>>> int create_file(char * base, int count, int max, int depth)
>>>>> {
>>>>> int i;
>>>>> for(i=0; i<count; i++) {
>>>>> char dir[256];
>>>>> int mydir = rand() % max;
>>>>> sprintf(dir, "%s/%d", path, mydir);
>>>>> if (depth >=1) {
>>>>> mkdir(dir,0777);
>>>>> create_dir(dir, count, max, depth - 1);
>>>>> } else {
>>>>> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
>>>>> printf("opened path : %s = %d\n", path, fd);
>>>>> close(fd);
>>>>> }
>>>>> }
>>>>> }
>>>>> int main(int argc, char argv[])
>>>>> {
>>>>> char path[256];
>>>>> while(1) {
>>>>> create_file("/import/SQL01", 1, 4 ,10);
>>>>> }
>>>>> }
>>>>>
>>>>
>>>> still don't see this behavior on 4.13 kernel. I suspect there is
>>>> something wrong with dentry lease. please check if your kernel
>>>> include:
>>>>
>>>> commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
>>>> commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
>>>>
>>>> The first commit can cause this issue, the second one fixes it.
>>>>
>>>> Regards
>>>> Yan, Zheng
>>>>
>>>>>
>>>>>
>>>>> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>>>>>> The workload behavior is like:
>>>>>>> mkdir DIR0
>>>>>>> mkdir DIR0/DIR1
>>>>>>> mkdir DIR0/DIR1/DIR2
>>>>>>> ....
>>>>>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>>>>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>>>>>
>>>>>>> and concurrently run on 50+ clients, the dir name in different
>>>>>>> clients may or maynot be the same.
>>>>>>>
>>>>>>> from the admin socket I was seeing ~50K create requests, but
>>>>>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>>>>>> so file create is slow.
>>>>>>>
>>>>>>> Where is the lookup comes from and can we have anyway to
>>>>>>> optimize it out ?
>>>>>>>
>>>>>>
>>>>>> I don't see this behavior when running following commands in 4.13
>>>>>> kernel client and luminous version ceph-fuse. which client do you use?
>>>>>>
>>>>>> mkdir d1
>>>>>> mkdir d1/d2
>>>>>> mkdir d1/d2/d3
>>>>>> mkdir d1/d2/d3/d4/
>>>>>> mkdir d1/d2/d3/d4/d5
>>>>>> touch d1/d2/d3/d4/d5/f
>>>>>>
>>>>>>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-23 0:44 ` Yan, Zheng
@ 2017-10-23 6:01 ` Xiaoxi Chen
2017-10-23 6:58 ` 陶冬冬
2017-10-23 7:41 ` Yan, Zheng
0 siblings, 2 replies; 11+ messages in thread
From: Xiaoxi Chen @ 2017-10-23 6:01 UTC (permalink / raw)
To: Yan, Zheng; +Cc: Ceph Development, John Spray
Yes, actually lots of (50+) clients are trying to create the same
large directory tree concurrently. So the behavior is most of the
mkdir will get -EEXISTS.
not very understand how the mkdir call from application level be
finally turned into lookup in MDS? could you please explain a bit
more ?
2017-10-23 8:44 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
> On Sun, Oct 22, 2017 at 11:27 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>> To add another data point, switched to ceph-fuse 12.2.0, still seeing
>> lots of lookup.
>> lookup avg 1892
>> mkdir avg 367
>> create avg 222
>> open avg 228
>>
>
> But in your test, mkdir avg was about 1.5 times of open avg. I think
> your test created millions of directories, lookups were from cache
> miss. You can try enlarging client_cache_size. But I don't think it
> will help much when active set of directory are so large.
>
>
>> 2017-10-21 2:09 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>>> @Zheng, my kernel doesn't even has c3f4688a08f. But 200fd27 ("ceph:
>>> use lookup request to revalidate dentry") is there.
>>>
>>> 2017-10-21 0:54 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>>>> Thanks, will check.
>>>>
>>>> A general question, does cephfs kernel client drop dentries/inode
>>>> cache aggressively? What I know is if MDS issue
>>>> CEPH_SESSION_RECALL_STATE, client will drop, but is there other cases
>>>> client will drop cache?
>>>>
>>>>
>>>>
>>>> 2017-10-20 16:39 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>> On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>>>>>>
>>>>>> I extract the logical of file creation in our workload into a
>>>>>> reproducer , like below.
>>>>>>
>>>>>> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
>>>>>> I thought the lookup is to open the directory tree so I tried to
>>>>>> pre-make most of the dirs , use ls -i trying to read the dentries and
>>>>>> cache it, then re-run the reproducer, seems nothing different..
>>>>>>
>>>>>> #include <sys/stat.h>
>>>>>> #include <fcntl.h>
>>>>>> int create_file(char * base, int count, int max, int depth)
>>>>>> {
>>>>>> int i;
>>>>>> for(i=0; i<count; i++) {
>>>>>> char dir[256];
>>>>>> int mydir = rand() % max;
>>>>>> sprintf(dir, "%s/%d", path, mydir);
>>>>>> if (depth >=1) {
>>>>>> mkdir(dir,0777);
>>>>>> create_dir(dir, count, max, depth - 1);
>>>>>> } else {
>>>>>> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
>>>>>> printf("opened path : %s = %d\n", path, fd);
>>>>>> close(fd);
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> int main(int argc, char argv[])
>>>>>> {
>>>>>> char path[256];
>>>>>> while(1) {
>>>>>> create_file("/import/SQL01", 1, 4 ,10);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>
>>>>> still don't see this behavior on 4.13 kernel. I suspect there is
>>>>> something wrong with dentry lease. please check if your kernel
>>>>> include:
>>>>>
>>>>> commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
>>>>> commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
>>>>>
>>>>> The first commit can cause this issue, the second one fixes it.
>>>>>
>>>>> Regards
>>>>> Yan, Zheng
>>>>>
>>>>>>
>>>>>>
>>>>>> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>>>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>>>>>>> The workload behavior is like:
>>>>>>>> mkdir DIR0
>>>>>>>> mkdir DIR0/DIR1
>>>>>>>> mkdir DIR0/DIR1/DIR2
>>>>>>>> ....
>>>>>>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>>>>>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>>>>>>
>>>>>>>> and concurrently run on 50+ clients, the dir name in different
>>>>>>>> clients may or maynot be the same.
>>>>>>>>
>>>>>>>> from the admin socket I was seeing ~50K create requests, but
>>>>>>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>>>>>>> so file create is slow.
>>>>>>>>
>>>>>>>> Where is the lookup comes from and can we have anyway to
>>>>>>>> optimize it out ?
>>>>>>>>
>>>>>>>
>>>>>>> I don't see this behavior when running following commands in 4.13
>>>>>>> kernel client and luminous version ceph-fuse. which client do you use?
>>>>>>>
>>>>>>> mkdir d1
>>>>>>> mkdir d1/d2
>>>>>>> mkdir d1/d2/d3
>>>>>>> mkdir d1/d2/d3/d4/
>>>>>>> mkdir d1/d2/d3/d4/d5
>>>>>>> touch d1/d2/d3/d4/d5/f
>>>>>>>
>>>>>>>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-23 6:01 ` Xiaoxi Chen
@ 2017-10-23 6:58 ` 陶冬冬
2017-10-23 7:41 ` Yan, Zheng
1 sibling, 0 replies; 11+ messages in thread
From: 陶冬冬 @ 2017-10-23 6:58 UTC (permalink / raw)
To: Xiaoxi Chen; +Cc: Yan, Zheng, Ceph Development, John Spray
Hi Xiaoxi ,
As i know, mkdir -p “/a/b/c/d/e”
client would first lookup the existed path , and do truly mkdir to the left part.
Thanks,
Dongdong.
> 在 2017年10月23日，下午2:01，Xiaoxi Chen <superdebuger@gmail.com> 写道：
>
> Yes, actually lots of (50+) clients are trying to create the same
> large directory tree concurrently. So the behavior is most of the
> mkdir will get -EEXISTS.
>
> not very understand how the mkdir call from application level be
> finally turned into lookup in MDS? could you please explain a bit
> more ?
>
> 2017-10-23 8:44 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>> On Sun, Oct 22, 2017 at 11:27 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>> To add another data point, switched to ceph-fuse 12.2.0, still seeing
>>> lots of lookup.
>>> lookup avg 1892
>>> mkdir avg 367
>>> create avg 222
>>> open avg 228
>>>
>>
>> But in your test, mkdir avg was about 1.5 times of open avg. I think
>> your test created millions of directories, lookups were from cache
>> miss. You can try enlarging client_cache_size. But I don't think it
>> will help much when active set of directory are so large.
>>
>>
>>> 2017-10-21 2:09 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>>>> @Zheng, my kernel doesn't even has c3f4688a08f. But 200fd27 ("ceph:
>>>> use lookup request to revalidate dentry") is there.
>>>>
>>>> 2017-10-21 0:54 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>>>>> Thanks, will check.
>>>>>
>>>>> A general question, does cephfs kernel client drop dentries/inode
>>>>> cache aggressively? What I know is if MDS issue
>>>>> CEPH_SESSION_RECALL_STATE, client will drop, but is there other cases
>>>>> client will drop cache?
>>>>>
>>>>>
>>>>>
>>>>> 2017-10-20 16:39 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>>> On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>>> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>>>>>>>
>>>>>>> I extract the logical of file creation in our workload into a
>>>>>>> reproducer , like below.
>>>>>>>
>>>>>>> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
>>>>>>> I thought the lookup is to open the directory tree so I tried to
>>>>>>> pre-make most of the dirs , use ls -i trying to read the dentries and
>>>>>>> cache it, then re-run the reproducer, seems nothing different..
>>>>>>>
>>>>>>> #include <sys/stat.h>
>>>>>>> #include <fcntl.h>
>>>>>>> int create_file(char * base, int count, int max, int depth)
>>>>>>> {
>>>>>>> int i;
>>>>>>> for(i=0; i<count; i++) {
>>>>>>> char dir[256];
>>>>>>> int mydir = rand() % max;
>>>>>>> sprintf(dir, "%s/%d", path, mydir);
>>>>>>> if (depth >=1) {
>>>>>>> mkdir(dir,0777);
>>>>>>> create_dir(dir, count, max, depth - 1);
>>>>>>> } else {
>>>>>>> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
>>>>>>> printf("opened path : %s = %d\n", path, fd);
>>>>>>> close(fd);
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> int main(int argc, char argv[])
>>>>>>> {
>>>>>>> char path[256];
>>>>>>> while(1) {
>>>>>>> create_file("/import/SQL01", 1, 4 ,10);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>
>>>>>> still don't see this behavior on 4.13 kernel. I suspect there is
>>>>>> something wrong with dentry lease. please check if your kernel
>>>>>> include:
>>>>>>
>>>>>> commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
>>>>>> commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
>>>>>>
>>>>>> The first commit can cause this issue, the second one fixes it.
>>>>>>
>>>>>> Regards
>>>>>> Yan, Zheng
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>>>>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>>>>>>>> The workload behavior is like:
>>>>>>>>> mkdir DIR0
>>>>>>>>> mkdir DIR0/DIR1
>>>>>>>>> mkdir DIR0/DIR1/DIR2
>>>>>>>>> ....
>>>>>>>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>>>>>>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>>>>>>>
>>>>>>>>> and concurrently run on 50+ clients, the dir name in different
>>>>>>>>> clients may or maynot be the same.
>>>>>>>>>
>>>>>>>>> from the admin socket I was seeing ~50K create requests, but
>>>>>>>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>>>>>>>> so file create is slow.
>>>>>>>>>
>>>>>>>>> Where is the lookup comes from and can we have anyway to
>>>>>>>>> optimize it out ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't see this behavior when running following commands in 4.13
>>>>>>>> kernel client and luminous version ceph-fuse. which client do you use?
>>>>>>>>
>>>>>>>> mkdir d1
>>>>>>>> mkdir d1/d2
>>>>>>>> mkdir d1/d2/d3
>>>>>>>> mkdir d1/d2/d3/d4/
>>>>>>>> mkdir d1/d2/d3/d4/d5
>>>>>>>> touch d1/d2/d3/d4/d5/f
>>>>>>>>
>>>>>>>>> Xiaoxi
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Huge lookup when recursively mkdir
2017-10-23 6:01 ` Xiaoxi Chen
2017-10-23 6:58 ` 陶冬冬
@ 2017-10-23 7:41 ` Yan, Zheng
1 sibling, 0 replies; 11+ messages in thread
From: Yan, Zheng @ 2017-10-23 7:41 UTC (permalink / raw)
To: Xiaoxi Chen; +Cc: Ceph Development, John Spray
On Mon, Oct 23, 2017 at 2:01 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
> Yes, actually lots of (50+) clients are trying to create the same
> large directory tree concurrently. So the behavior is most of the
> mkdir will get -EEXISTS.
>
> not very understand how the mkdir call from application level be
> finally turned into lookup in MDS? could you please explain a bit
> more ?
For 'mkdir a'. kernel first does 'lookup a'. if 'a' does not exists,
does real mkdir, otherwise return -EEXIST.
>
> 2017-10-23 8:44 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>> On Sun, Oct 22, 2017 at 11:27 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>> To add another data point, switched to ceph-fuse 12.2.0, still seeing
>>> lots of lookup.
>>> lookup avg 1892
>>> mkdir avg 367
>>> create avg 222
>>> open avg 228
>>>
>>
>> But in your test, mkdir avg was about 1.5 times of open avg. I think
>> your test created millions of directories, lookups were from cache
>> miss. You can try enlarging client_cache_size. But I don't think it
>> will help much when active set of directory are so large.
>>
>>
>>> 2017-10-21 2:09 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>>>> @Zheng, my kernel doesn't even has c3f4688a08f. But 200fd27 ("ceph:
>>>> use lookup request to revalidate dentry") is there.
>>>>
>>>> 2017-10-21 0:54 GMT+08:00 Xiaoxi Chen <superdebuger@gmail.com>:
>>>>> Thanks, will check.
>>>>>
>>>>> A general question, does cephfs kernel client drop dentries/inode
>>>>> cache aggressively? What I know is if MDS issue
>>>>> CEPH_SESSION_RECALL_STATE, client will drop, but is there other cases
>>>>> client will drop cache?
>>>>>
>>>>>
>>>>>
>>>>> 2017-10-20 16:39 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>>> On Fri, Oct 20, 2017 at 3:28 PM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>>> Centos 7.3, kernel version 3.10.0-514.26.2.el7.x86_64.
>>>>>>>
>>>>>>> I extract the logical of file creation in our workload into a
>>>>>>> reproducer , like below.
>>>>>>>
>>>>>>> Concurrently run the reproducer in 2+ node can see a lots of lookup OP.
>>>>>>> I thought the lookup is to open the directory tree so I tried to
>>>>>>> pre-make most of the dirs , use ls -i trying to read the dentries and
>>>>>>> cache it, then re-run the reproducer, seems nothing different..
>>>>>>>
>>>>>>> #include <sys/stat.h>
>>>>>>> #include <fcntl.h>
>>>>>>> int create_file(char * base, int count, int max, int depth)
>>>>>>> {
>>>>>>> int i;
>>>>>>> for(i=0; i<count; i++) {
>>>>>>> char dir[256];
>>>>>>> int mydir = rand() % max;
>>>>>>> sprintf(dir, "%s/%d", path, mydir);
>>>>>>> if (depth >=1) {
>>>>>>> mkdir(dir,0777);
>>>>>>> create_dir(dir, count, max, depth - 1);
>>>>>>> } else {
>>>>>>> int fd = open(dir, O_CREAT | O_EXCL| O_WRONLY , 0666);
>>>>>>> printf("opened path : %s = %d\n", path, fd);
>>>>>>> close(fd);
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> int main(int argc, char argv[])
>>>>>>> {
>>>>>>> char path[256];
>>>>>>> while(1) {
>>>>>>> create_file("/import/SQL01", 1, 4 ,10);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>
>>>>>> still don't see this behavior on 4.13 kernel. I suspect there is
>>>>>> something wrong with dentry lease. please check if your kernel
>>>>>> include:
>>>>>>
>>>>>> commit c3f4688a08f (ceph: don't set req->r_locked_dir in ceph_d_revalidate)
>>>>>> commit 5eb9f6040f3 (ceph: do a LOOKUP in d_revalidate instead of GETATTR)
>>>>>>
>>>>>> The first commit can cause this issue, the second one fixes it.
>>>>>>
>>>>>> Regards
>>>>>> Yan, Zheng
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2017-10-20 10:55 GMT+08:00 Yan, Zheng <ukernel@gmail.com>:
>>>>>>>> On Fri, Oct 20, 2017 at 12:49 AM, Xiaoxi Chen <superdebuger@gmail.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am seeing a lot of lookup request when doing recursive mkdir.
>>>>>>>>> The workload behavior is like:
>>>>>>>>> mkdir DIR0
>>>>>>>>> mkdir DIR0/DIR1
>>>>>>>>> mkdir DIR0/DIR1/DIR2
>>>>>>>>> ....
>>>>>>>>> mkdir DIR0/DIR1/DIR2......./DIR7
>>>>>>>>> create DIR0/DIR1/DIR2......./DIR7/FILE1
>>>>>>>>>
>>>>>>>>> and concurrently run on 50+ clients, the dir name in different
>>>>>>>>> clients may or maynot be the same.
>>>>>>>>>
>>>>>>>>> from the admin socket I was seeing ~50K create requests, but
>>>>>>>>> got 400K lookup requests. The lookup eat up most of the mds capability
>>>>>>>>> so file create is slow.
>>>>>>>>>
>>>>>>>>> Where is the lookup comes from and can we have anyway to
>>>>>>>>> optimize it out ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't see this behavior when running following commands in 4.13
>>>>>>>> kernel client and luminous version ceph-fuse. which client do you use?
>>>>>>>>
>>>>>>>> mkdir d1
>>>>>>>> mkdir d1/d2
>>>>>>>> mkdir d1/d2/d3
>>>>>>>> mkdir d1/d2/d3/d4/
>>>>>>>> mkdir d1/d2/d3/d4/d5
>>>>>>>> touch d1/d2/d3/d4/d5/f
>>>>>>>>
>>>>>>>>> Xiaoxi
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2017-10-23 7:41 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-10-19 16:49 Huge lookup when recursively mkdir Xiaoxi Chen
2017-10-20 2:55 ` Yan, Zheng
2017-10-20 7:28 ` Xiaoxi Chen
2017-10-20 8:39 ` Yan, Zheng
2017-10-20 16:54 ` Xiaoxi Chen
2017-10-20 18:09 ` Xiaoxi Chen
2017-10-22 15:27 ` Xiaoxi Chen
2017-10-23 0:44 ` Yan, Zheng
2017-10-23 6:01 ` Xiaoxi Chen
2017-10-23 6:58 ` 陶冬冬
2017-10-23 7:41 ` Yan, Zheng
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.