Re: ceph-fs tests

[Smart Weblications GmbH - Florian Wiessner <f.wiessner@smart-weblications.de>]

Am 05.09.2012 18:22, schrieb Tommi Virtanen:
> On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian
> Wiessner <f.wiessner@smart-weblications.de> wrote:
>> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs.
>>
>> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few
>> times, then stopping one osd (osd.1), afer a while of recovering, suddenly:

> 
> Please provide English error messages when you share things with the
> list. In this case I can figure out what the message is, but really,
> we're all pattern matching animals and the specific strings in
> /usr/include/asm-generic/errno.h are what we know.
> 

OK, will change locales.

>> no space left on device, but:
>>
>> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean,
>> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail;
>> 36677/215076 degraded (17.053%)
>>
>> there is space left?
> 
> Only 844 MB available, with the pseudo-random placement policies,
> means you practically are out of space.
> 
> It looks like you had only 15GB to begin with, and with typical
> replication, that's <5GB usable space. That is dangerously small for
> any real use; Ceph currently does not cope very well with running out
> of space.
> 

It is a test-cluster running on my thinkpad, its main purpose is to test cephfs,
there is no need for real space. I added osd.1 again, then after recovery the
problem went away. I forced this situation to check how cephfs will behave when
cluster is near-full, osd fails and ceph tries to recover until backfill fills
up other osds so ceph is full.

I observed on the client that no IO was possible anymore so that the client was
unusable.

Is there a smarter way to handle this? It is bad that cephfs then stalls, it
would be better if it just returns that there is no space left, but still allow
read access... can this be tuned somewhere?





-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html