zero size file on cephfs

zero size file on cephfs

[Drunkard Zhang]

Hi,
    I'm using cephfs for a while, it's great, but I found a serious problem now.

My situation is simple: a lot of servers receive syslog to local disk,
then mount ceph FS via kernel driver and move those received files to
ceph FS.

Some days ago, I found my script missed some file while processing
data, then I noticed those files' size is zero. But it's impossible,
rsyslog won't generate file until data arrives, so I logged the file
size before mv, so here's one example from these exceptions:

In log:
2013-11-07 18:17:29 /scripts/log/collector: file size: 4910689
/gwbn/nat/20131107/dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log

Zero size file on ceph FS:
-rw-r--r-- 1 root root     0 Jan  1  1970
dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log

stat dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log
  File: ‘dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log’
  Size: 0             Blocks: 0          IO Block: 4194304 regular empty file
Device: 0h/0d    Inode: 1099518968218  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-11-07 17:00:00.167501860 +0800
Modify: 1970-01-01 08:00:00.000000000 +0800
Change: 2013-11-07 18:17:29.843558000 +0800
 Birth: -

I'm running multiple versions of kernel, scales from 3.8.x to 3.11.6.
Most of the zero file event occurs on those servers with kernel
version of 3.10.[45], some on 3.8.6 and 3.11.4, I didn't tested all of
them, these is my running versions.

I think the ceph FS kernel driver should sync before umount, isn't it
? So what can I do to shoot this problem down ? And any advice of
temparory solution to avoid this, manually sync?
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: zero size file on cephfs

[Yan, Zheng]

sounds like caused by the file recover bug. The fix
(https://github.com/ceph/ceph/commit/eb381ffc8db14f13a7c5e3528a109bf89a7c5b31)
is included in the 0.72 release.

Regards
Yan, Zheng


On Fri, Nov 8, 2013 at 10:36 AM, Drunkard Zhang <gongfan193@gmail.com> wrote:
> Hi,
>     I'm using cephfs for a while, it's great, but I found a serious problem now.
>
> My situation is simple: a lot of servers receive syslog to local disk,
> then mount ceph FS via kernel driver and move those received files to
> ceph FS.
>
> Some days ago, I found my script missed some file while processing
> data, then I noticed those files' size is zero. But it's impossible,
> rsyslog won't generate file until data arrives, so I logged the file
> size before mv, so here's one example from these exceptions:
>
> In log:
> 2013-11-07 18:17:29 /scripts/log/collector: file size: 4910689
> /gwbn/nat/20131107/dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log
>
> Zero size file on ceph FS:
> -rw-r--r-- 1 root root     0 Jan  1  1970
> dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log
>
> stat dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log
>   File: ‘dnsquery-pb24-192.168.156.60-by_squid79-20131107_17.log’
>   Size: 0             Blocks: 0          IO Block: 4194304 regular empty file
> Device: 0h/0d    Inode: 1099518968218  Links: 1
> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2013-11-07 17:00:00.167501860 +0800
> Modify: 1970-01-01 08:00:00.000000000 +0800
> Change: 2013-11-07 18:17:29.843558000 +0800
>  Birth: -
>
> I'm running multiple versions of kernel, scales from 3.8.x to 3.11.6.
> Most of the zero file event occurs on those servers with kernel
> version of 3.10.[45], some on 3.8.6 and 3.11.4, I didn't tested all of
> them, these is my running versions.
>
> I think the ceph FS kernel driver should sync before umount, isn't it
> ? So what can I do to shoot this problem down ? And any advice of
> temparory solution to avoid this, manually sync?
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html