Re: CephFS Space Accounting and Quotas

From: "Jim Schutt" <jaschut@sandia.gov>
To: Greg Farnum <greg@inktank.com>
Cc: ceph-devel@vger.kernel.org, Sage Weil <sage@inktank.com>,
	Wido den Hollander <wido@42on.com>
Subject: Re: CephFS Space Accounting and Quotas
Date: Mon, 11 Mar 2013 08:47:31 -0600	[thread overview]
Message-ID: <513DEE83.8030909@sandia.gov> (raw)
In-Reply-To: <391D23EA41AE4F05BCE2E6367BA0FC3C@inktank.com>

On 03/08/2013 07:05 PM, Greg Farnum wrote:
> On Friday, March 8, 2013 at 2:45 PM, Jim Schutt wrote:
>> On 03/07/2013 08:15 AM, Jim Schutt wrote:
>>> On 03/06/2013 05:18 PM, Greg Farnum wrote:
>>>> On Wednesday, March 6, 2013 at 3:14 PM, Jim Schutt wrote:
>>>  
>>  
>>  
>>  
>> [snip]
>>  
>>>>> Do you want the MDS log at 10 or 20?
>>>> More is better. ;)
>>>  
>>>  
>>>  
>>> OK, thanks.
>>  
>> I've sent some mds logs via private email...
>>  
>> -- Jim  
> I'm going to need to probe into this a bit more, but on an initial
> examination I see that most of your stats are actually happening very
> quickly — it's just that occasionally they take quite a while.

Interesting...

> Going
> through the MDS log for one of those, the inode in question is
> flagged with "needsrecover" from its first appearance in the log —
> that really shouldn't happen unless a client had write caps on it and
> the client disappeared. Any ideas? The slowness is being caused by
> the MDS going out and looking at every object which could be in the
> file — there are a lot since the file has a listed size of 8GB.

For this run, the MDS logging slowed it down enough to cause the
client caps to occasionally go stale.  I don't think it's the cause
of the issue, because I was having it before I turned MDS debugging
up.  My client caps never go stale at, e.g., debug mds 5.

Otherwise, there were no signs of trouble while writing the files.

Can you suggest which kernel client debugging I might enable that
would help understand what is happening?  Also, I have the full
MDS log from writing the files, if that will help.  It's big (~10 GiB).

> (There are several other mysteries here that can probably be traced
> to different varieties of non-optimal and buggy code as well — there
> is a client which has write caps on the inode in question despite it
> needing recovery, but the recovery isn't triggered until the stat
> event occurs, etc).

OK, thanks for taking a look.  Let me know if there is other
logging I can enable that will be helpful.

-- Jim

> -Greg
> 
> Software Engineer #42 @ http://inktank.com | http://ceph.com 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html