Re: scubbing for a long time and not finished

Re: scubbing for a long time and not finished

[Xinze Chi]

Currently, users do not know  when some pg do scrubbing for a long time.
I think whether we could give some warming if it happend (defined as
osd_scrub_max_time).
It would tell the user something may be wrong in cluster.


2015-03-17 21:21 GMT+08:00 池信泽 <xmdxcxz@gmail.com>:
>
> On 周二, 3月 17, 2015 at 10:01 上午, Xinze Chi <xmdxcxz@gmail.com> wrote:
>
> hi,all:
>
>     I find a pg on my test cluster in doing scrubbing for a long time
> and not finish. there are not some useful scrubbing log. scrubs_active
> is 1, so inc_scrubs_pending return false. I think the reason is that
> some scrub message is lost, so primary can not continue chunky_scrub ,
> so it hang up at scrubbing.
>
>    Could anyone give some suggestion?
>
>    Thanks
>
>
> [root@ceph0 ~]# date
> Tue Mar 17 09:54:54 CST 2015
> [root@ceph0 ~]# ceph pg dump | grep scrub
> dumped all in format plain
> pg_stat objects mip degr misp unf bytes log disklog state state_stamp
> v reported up up_primary acting acting_primary last_scrub
> scrub_stamplast_deep_scrub deep_scrub_stamp
> 1.97 30 0 0 0 0 117702656 31 31 active+clean+scrubbing 2015-03-16
> 14:50:02.110796 78'31 78:50 [9,6,1] 9 [9,6,1] 9 0'0 2015-03-15
> 14:49:33.661597 0'0 2015-03-13 14:48:53.341679
>
>    The attachment is the log from primary, the scrubbing pg is 1.97s0.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: scubbing for a long time and not finished

[Sage Weil]

On Thu, 19 Mar 2015, Xinze Chi wrote:
> Currently, users do not know  when some pg do scrubbing for a long time.
> I think whether we could give some warming if it happend (defined as
> osd_scrub_max_time).
> It would tell the user something may be wrong in cluster.

This should be pretty straightforward to add along with the other "stuck 
x" warnings based on the pg_stat_t state timestamps.  On the otherhead, 
that may be a somewhat heavyweight approach (each new warning bloats the 
stat structure a bit); open to other ideas!

sage

Re: scubbing for a long time and not finished

[Xinze Chi]

I add a issue http://tracker.ceph.com/issues/11162.

2015-03-19 21:33 GMT+08:00 Sage Weil <sage@newdream.net>:
> On Thu, 19 Mar 2015, Xinze Chi wrote:
>> Currently, users do not know  when some pg do scrubbing for a long time.
>> I think whether we could give some warming if it happend (defined as
>> osd_scrub_max_time).
>> It would tell the user something may be wrong in cluster.
>
> This should be pretty straightforward to add along with the other "stuck
> x" warnings based on the pg_stat_t state timestamps.  On the otherhead,
> that may be a somewhat heavyweight approach (each new warning bloats the
> stat structure a bit); open to other ideas!
>
> sage