ceph-fs tests

ceph-fs tests

[Smart Weblications GmbH - Florian Wiessner]

hi,


i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs.

i mount ceph via fuse, then i downloaded kernel tree and decompressed a few
times, then stopping one osd (osd.1), afer a while of recovering, suddenly:

tar: linux-3.5.3/drivers/media/video/zoran/zr36060.h: Kann write nicht
ausführen: Auf dem Gerät ist kein Speicherplatz mehr verfügbar
linux-3.5.3/drivers/media/video/zr364xx.c
tar: linux-3.5.3/drivers/media/video/zr364xx.c: Kann write nicht ausführen: Auf
dem Gerät ist kein Speicherplatz mehr verfügbar
linux-3.5.3/drivers/memory/

no space left on device, but:

2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean,
64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail;
36677/215076 degraded (17.053%)

there is space left?
-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

rbd 0.48 storage support for kvm proxmox distribution available

[Alexandre DERUMIER]

Hi List,

We have added rbd 0.48 support to the proxmox 2.1 kvm distribution
http://www.proxmox.com/products/proxmox-ve


Proxmox setup:

edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) 

rbd: mycephcluster
       monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789
       pool rbd
       username admin
       authsupported cephx;none
       content images 


then you need to copy the keyring file from ceph to proxmox

scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring 



For now, you can add/delete/resize rbd volumes from gui.
Snapshots/cloning will be added soon (when layering will be available)


Regards,

Alexandre Derumier

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Wido den Hollander]

On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote:
> Hi List,
>
> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution
> http://www.proxmox.com/products/proxmox-ve
>
>
> Proxmox setup:
>
> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet)
>
> rbd: mycephcluster
>         monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789
>         pool rbd
>         username admin
>         authsupported cephx;none
>         content images
>

Proxmox doesn't use libvirt, does it?

Any plans to implement the RBD caching?

Nice work though!

Wido

>
> then you need to copy the keyring file from ceph to proxmox
>
> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring
>
>
>
> For now, you can add/delete/resize rbd volumes from gui.
> Snapshots/cloning will be added soon (when layering will be available)
>
>
> Regards,
>
> Alexandre Derumier
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Alexandre DERUMIER]

>>Proxmox doesn't use libvirt, does it? 
Yes, we don't use libvirt. 

>>Any plans to implement the RBD caching? 

It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days)

Tunning of cache size can be done with a /etc/ceph.conf file



----- Mail original ----- 

De: "Wido den Hollander" <wido@widodh.nl> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Mercredi 5 Septembre 2012 11:11:27 
Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available 

On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote: 
> Hi List, 
> 
> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution 
> http://www.proxmox.com/products/proxmox-ve 
> 
> 
> Proxmox setup: 
> 
> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) 
> 
> rbd: mycephcluster 
> monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 
> pool rbd 
> username admin 
> authsupported cephx;none 
> content images 
> 

Proxmox doesn't use libvirt, does it? 

Any plans to implement the RBD caching? 

Nice work though! 

Wido 

> 
> then you need to copy the keyring file from ceph to proxmox 
> 
> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring 
> 
> 
> 
> For now, you can add/delete/resize rbd volumes from gui. 
> Snapshots/cloning will be added soon (when layering will be available) 
> 
> 
> Regards, 
> 
> Alexandre Derumier 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Wido den Hollander]

On 09/05/2012 11:30 AM, Alexandre DERUMIER wrote:
>>> Proxmox doesn't use libvirt, does it?
> Yes, we don't use libvirt.
>
>>> Any plans to implement the RBD caching?
>
> It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days)
>
> Tunning of cache size can be done with a /etc/ceph.conf file
>

That is kind of dangerous imho and for a couple of reasons.

For configuring the storage you have /etc/pve/storage.cfg where you can 
add the RBD pool, configure the monitors and cephx, but for caching you 
rely in librbd reading ceph.conf?

That is hidden from the user, reading /etc/ceph/ceph.conf will go 
without your knowledge. I'd opt for passing down all the options to Qemu 
and being able to run without a ceph.conf

I've ran into the same problems with libvirt and CloudStack. I couldn't 
figure out why libvirt was still able to connect to a specific cluster 
until I found out my ceph.conf was still in place.

I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by 
default with librbd/librados to take away these kind of issues.

And you would also have this weird situation where the ceph.conf could 
have a couple of monitor entries and your "storage.cfg", how will that 
work out?

I would try not to rely on the ceph.conf at all and have Proxmox pass 
all the configuration options down to Qemu.

Wido

>
>
> ----- Mail original -----
>
> De: "Wido den Hollander" <wido@widodh.nl>
> À: "Alexandre DERUMIER" <aderumier@odiso.com>
> Cc: ceph-devel@vger.kernel.org
> Envoyé: Mercredi 5 Septembre 2012 11:11:27
> Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available
>
> On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote:
>> Hi List,
>>
>> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution
>> http://www.proxmox.com/products/proxmox-ve
>>
>>
>> Proxmox setup:
>>
>> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet)
>>
>> rbd: mycephcluster
>> monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789
>> pool rbd
>> username admin
>> authsupported cephx;none
>> content images
>>
>
> Proxmox doesn't use libvirt, does it?
>
> Any plans to implement the RBD caching?
>
> Nice work though!
>
> Wido
>
>>
>> then you need to copy the keyring file from ceph to proxmox
>>
>> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring
>>
>>
>>
>> For now, you can add/delete/resize rbd volumes from gui.
>> Snapshots/cloning will be added soon (when layering will be available)
>>
>>
>> Regards,
>>
>> Alexandre Derumier
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Alexandre DERUMIER]

Thanks for the infos.

I dind't have documented the ceph.conf feature,it's more a workaround for now for users which want to tune some values.

Indeed,I'm planning to add the cache size options to the proxmox storage.cfg.


Thanks

Alexandre


----- Mail original ----- 

De: "Wido den Hollander" <wido@widodh.nl> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Mercredi 5 Septembre 2012 14:35:16 
Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available 

On 09/05/2012 11:30 AM, Alexandre DERUMIER wrote: 
>>> Proxmox doesn't use libvirt, does it? 
> Yes, we don't use libvirt. 
> 
>>> Any plans to implement the RBD caching? 
> 
> It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days) 
> 
> Tunning of cache size can be done with a /etc/ceph.conf file 
> 

That is kind of dangerous imho and for a couple of reasons. 

For configuring the storage you have /etc/pve/storage.cfg where you can 
add the RBD pool, configure the monitors and cephx, but for caching you 
rely in librbd reading ceph.conf? 

That is hidden from the user, reading /etc/ceph/ceph.conf will go 
without your knowledge. I'd opt for passing down all the options to Qemu 
and being able to run without a ceph.conf 

I've ran into the same problems with libvirt and CloudStack. I couldn't 
figure out why libvirt was still able to connect to a specific cluster 
until I found out my ceph.conf was still in place. 

I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by 
default with librbd/librados to take away these kind of issues. 

And you would also have this weird situation where the ceph.conf could 
have a couple of monitor entries and your "storage.cfg", how will that 
work out? 

I would try not to rely on the ceph.conf at all and have Proxmox pass 
all the configuration options down to Qemu. 

Wido 

> 
> 
> ----- Mail original ----- 
> 
> De: "Wido den Hollander" <wido@widodh.nl> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org 
> Envoyé: Mercredi 5 Septembre 2012 11:11:27 
> Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available 
> 
> On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote: 
>> Hi List, 
>> 
>> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution 
>> http://www.proxmox.com/products/proxmox-ve 
>> 
>> 
>> Proxmox setup: 
>> 
>> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) 
>> 
>> rbd: mycephcluster 
>> monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 
>> pool rbd 
>> username admin 
>> authsupported cephx;none 
>> content images 
>> 
> 
> Proxmox doesn't use libvirt, does it? 
> 
> Any plans to implement the RBD caching? 
> 
> Nice work though! 
> 
> Wido 
> 
>> 
>> then you need to copy the keyring file from ceph to proxmox 
>> 
>> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring 
>> 
>> 
>> 
>> For now, you can add/delete/resize rbd volumes from gui. 
>> Snapshots/cloning will be added soon (when layering will be available) 
>> 
>> 
>> Regards, 
>> 
>> Alexandre Derumier 
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
>> the body of a message to majordomo@vger.kernel.org 
>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Josh Durgin]

On Wed, 05 Sep 2012 14:35:16 +0200, Wido den Hollander <wido@widodh.nl>
wrote:
> On 09/05/2012 11:30 AM, Alexandre DERUMIER wrote:
>>>> Proxmox doesn't use libvirt, does it?
>> Yes, we don't use libvirt.
>>
>>>> Any plans to implement the RBD caching?
>>
>> It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days)
>>
>> Tunning of cache size can be done with a /etc/ceph.conf file
>>
> 
> That is kind of dangerous imho and for a couple of reasons.
> 
> For configuring the storage you have /etc/pve/storage.cfg where you
> can add the RBD pool, configure the monitors and cephx, but for
> caching you rely in librbd reading ceph.conf?
> 
> That is hidden from the user, reading /etc/ceph/ceph.conf will go
> without your knowledge. I'd opt for passing down all the options to
> Qemu and being able to run without a ceph.conf
> 
> I've ran into the same problems with libvirt and CloudStack. I
> couldn't figure out why libvirt was still able to connect to a
> specific cluster until I found out my ceph.conf was still in place.
> 
> I also thought it is on the roadmap to not read /etc/ceph/ceph.conf
> by default with librbd/librados to take away these kind of issues.

I don't think we'll want to change the default behavior (qemu reading
/etc/ceph/ceph.conf) for backwards compatibility, but I agree that we
should avoid relying on it in the future.

Josh

> And you would also have this weird situation where the ceph.conf
> could have a couple of monitor entries and your "storage.cfg", how
> will that work out?
> 
> I would try not to rely on the ceph.conf at all and have Proxmox pass
> all the configuration options down to Qemu.
> 
> Wido
> 
>>
>>
>> ----- Mail original -----
>>
>> De: "Wido den Hollander" <wido@widodh.nl>
>> À: "Alexandre DERUMIER" <aderumier@odiso.com>
>> Cc: ceph-devel@vger.kernel.org
>> Envoyé: Mercredi 5 Septembre 2012 11:11:27
>> Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available
>>
>> On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote:
>>> Hi List,
>>>
>>> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution
>>> http://www.proxmox.com/products/proxmox-ve
>>>
>>>
>>> Proxmox setup:
>>>
>>> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet)
>>>
>>> rbd: mycephcluster
>>> monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789
>>> pool rbd
>>> username admin
>>> authsupported cephx;none
>>> content images
>>>
>>
>> Proxmox doesn't use libvirt, does it?
>>
>> Any plans to implement the RBD caching?
>>
>> Nice work though!
>>
>> Wido
>>
>>>
>>> then you need to copy the keyring file from ceph to proxmox
>>>
>>> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring
>>>
>>>
>>>
>>> For now, you can add/delete/resize rbd volumes from gui.
>>> Snapshots/cloning will be added soon (when layering will be available)
>>>
>>>
>>> Regards,
>>>
>>> Alexandre Derumier
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-fs tests

[Tommi Virtanen]

On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian
Wiessner <f.wiessner@smart-weblications.de> wrote:
> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs.
>
> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few
> times, then stopping one osd (osd.1), afer a while of recovering, suddenly:
>
> tar: linux-3.5.3/drivers/media/video/zoran/zr36060.h: Kann write nicht
> ausführen: Auf dem Gerät ist kein Speicherplatz mehr verfügbar
> linux-3.5.3/drivers/media/video/zr364xx.c
> tar: linux-3.5.3/drivers/media/video/zr364xx.c: Kann write nicht ausführen: Auf
> dem Gerät ist kein Speicherplatz mehr verfügbar
> linux-3.5.3/drivers/memory/

Please provide English error messages when you share things with the
list. In this case I can figure out what the message is, but really,
we're all pattern matching animals and the specific strings in
/usr/include/asm-generic/errno.h are what we know.

> no space left on device, but:
>
> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean,
> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail;
> 36677/215076 degraded (17.053%)
>
> there is space left?

Only 844 MB available, with the pseudo-random placement policies,
means you practically are out of space.

It looks like you had only 15GB to begin with, and with typical
replication, that's <5GB usable space. That is dangerously small for
any real use; Ceph currently does not cope very well with running out
of space.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-fs tests

[Gregory Farnum]

On Wed, Sep 5, 2012 at 9:22 AM, Tommi Virtanen <tv@inktank.com> wrote:
> On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian
> Wiessner <f.wiessner@smart-weblications.de> wrote:
>> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs.
>>
>> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few
>> times, then stopping one osd (osd.1), afer a while of recovering, suddenly:
>>
>> tar: linux-3.5.3/drivers/media/video/zoran/zr36060.h: Kann write nicht
>> ausführen: Auf dem Gerät ist kein Speicherplatz mehr verfügbar
>> linux-3.5.3/drivers/media/video/zr364xx.c
>> tar: linux-3.5.3/drivers/media/video/zr364xx.c: Kann write nicht ausführen: Auf
>> dem Gerät ist kein Speicherplatz mehr verfügbar
>> linux-3.5.3/drivers/memory/
>
> Please provide English error messages when you share things with the
> list. In this case I can figure out what the message is, but really,
> we're all pattern matching animals and the specific strings in
> /usr/include/asm-generic/errno.h are what we know.
>
>> no space left on device, but:
>>
>> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean,
>> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail;
>> 36677/215076 degraded (17.053%)
>>
>> there is space left?
>
> Only 844 MB available, with the pseudo-random placement policies,
> means you practically are out of space.
>
> It looks like you had only 15GB to begin with, and with typical
> replication, that's <5GB usable space. That is dangerously small for
> any real use; Ceph currently does not cope very well with running out
> of space.

In this particular case, one of the OSDs is more than 95% full and has
been marked as full (which stops cluster IO) to prevent those
catastrophic failures from occurring. If you look at the full output
of ceph -s, you should see a warning about having OSDs full and
near-full.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Tommi Virtanen]

On Wed, Sep 5, 2012 at 5:35 AM, Wido den Hollander <wido@widodh.nl> wrote:
> I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by
> default with librbd/librados to take away these kind of issues.

Hmm. I'm not intimately familiar with librbd, but it seems it just
takes RADOS ioctx as an argument, and as far as I know (/can
experiment in 2 minutes) you definitely have a choice of whether
librados reads a config file or not..

rbd the command line tool does read a config file by default; pass
--conf=/dev/null to prevent that, then pass what ever config options
you need to set as --foo-bar=baz.

Re: ceph-fs tests

[Smart Weblications GmbH - Florian Wiessner]

Am 05.09.2012 18:22, schrieb Tommi Virtanen:
> On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian
> Wiessner <f.wiessner@smart-weblications.de> wrote:
>> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs.
>>
>> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few
>> times, then stopping one osd (osd.1), afer a while of recovering, suddenly:

> 
> Please provide English error messages when you share things with the
> list. In this case I can figure out what the message is, but really,
> we're all pattern matching animals and the specific strings in
> /usr/include/asm-generic/errno.h are what we know.
> 

OK, will change locales.

>> no space left on device, but:
>>
>> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean,
>> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail;
>> 36677/215076 degraded (17.053%)
>>
>> there is space left?
> 
> Only 844 MB available, with the pseudo-random placement policies,
> means you practically are out of space.
> 
> It looks like you had only 15GB to begin with, and with typical
> replication, that's <5GB usable space. That is dangerously small for
> any real use; Ceph currently does not cope very well with running out
> of space.
> 

It is a test-cluster running on my thinkpad, its main purpose is to test cephfs,
there is no need for real space. I added osd.1 again, then after recovery the
problem went away. I forced this situation to check how cephfs will behave when
cluster is near-full, osd fails and ceph tries to recover until backfill fills
up other osds so ceph is full.

I observed on the client that no IO was possible anymore so that the client was
unusable.

Is there a smarter way to handle this? It is bad that cephfs then stalls, it
would be better if it just returns that there is no space left, but still allow
read access... can this be tuned somewhere?





-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Tommi Virtanen]

On Wed, Sep 5, 2012 at 9:40 AM, Tommi Virtanen <tv@inktank.com> wrote:
> On Wed, Sep 5, 2012 at 5:35 AM, Wido den Hollander <wido@widodh.nl> wrote:
>> I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by
>> default with librbd/librados to take away these kind of issues.
>
> Hmm. I'm not intimately familiar with librbd, but it seems it just
> takes RADOS ioctx as an argument, and as far as I know (/can
> experiment in 2 minutes) you definitely have a choice of whether
> librados reads a config file or not..
>
> rbd the command line tool does read a config file by default; pass
> --conf=/dev/null to prevent that, then pass what ever config options
> you need to set as --foo-bar=baz.

So *QEmu* makes librados read a config file.

You can avoid that just like with the rbd command line tool, pass
conf=/dev/null. Quoting
git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git block/rbd.c:

 * The "conf" option specifies a Ceph configuration file to read.  If
 * it is not specified, we will read from the default Ceph locations
 * (e.g., /etc/ceph/ceph.conf).  To avoid reading _any_ configuration
 * file, specify conf=/dev/null.

Re: rbd 0.48 storage support for kvm proxmox distribution available

[Josh Durgin]

On 09/05/2012 09:40 AM, Tommi Virtanen wrote:
> On Wed, Sep 5, 2012 at 5:35 AM, Wido den Hollander <wido@widodh.nl> wrote:
>> I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by
>> default with librbd/librados to take away these kind of issues.
>
> Hmm. I'm not intimately familiar with librbd, but it seems it just
> takes RADOS ioctx as an argument, and as far as I know (/can
> experiment in 2 minutes) you definitely have a choice of whether
> librados reads a config file or not..
>
> rbd the command line tool does read a config file by default; pass
> --conf=/dev/null to prevent that, then pass what ever config options
> you need to set as --foo-bar=baz.

That's right, all the configuration is done by calls to the
rados_conf_* functions [1] before calling rados_connect. It
looks like the libvirt storage pool doesn't call
rados_conf_read_file(cluster, NULL), so it won't read /etc/ceph/ceph.conf.

Josh

[1] http://ceph.com/docs/master/api/librados/#rados_conf_read_file

Re: ceph-fs tests

[Gregory Farnum]

On Wed, Sep 5, 2012 at 9:42 AM, Smart Weblications GmbH - Florian
Wiessner <f.wiessner@smart-weblications.de> wrote:
> Am 05.09.2012 18:22, schrieb Tommi Virtanen:
>> On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian
>> Wiessner <f.wiessner@smart-weblications.de> wrote:
>>> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs.
>>>
>>> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few
>>> times, then stopping one osd (osd.1), afer a while of recovering, suddenly:
>
>>
>> Please provide English error messages when you share things with the
>> list. In this case I can figure out what the message is, but really,
>> we're all pattern matching animals and the specific strings in
>> /usr/include/asm-generic/errno.h are what we know.
>>
>
> OK, will change locales.
>
>>> no space left on device, but:
>>>
>>> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean,
>>> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail;
>>> 36677/215076 degraded (17.053%)
>>>
>>> there is space left?
>>
>> Only 844 MB available, with the pseudo-random placement policies,
>> means you practically are out of space.
>>
>> It looks like you had only 15GB to begin with, and with typical
>> replication, that's <5GB usable space. That is dangerously small for
>> any real use; Ceph currently does not cope very well with running out
>> of space.
>>
>
> It is a test-cluster running on my thinkpad, its main purpose is to test cephfs,
> there is no need for real space. I added osd.1 again, then after recovery the
> problem went away. I forced this situation to check how cephfs will behave when
> cluster is near-full, osd fails and ceph tries to recover until backfill fills
> up other osds so ceph is full.
>
> I observed on the client that no IO was possible anymore so that the client was
> unusable.
>
> Is there a smarter way to handle this? It is bad that cephfs then stalls, it
> would be better if it just returns that there is no space left, but still allow
> read access... can this be tuned somewhere?

What client were you using? I believe it does allow reads while full —
but your client can pretty easily get itself into a situation where it
needs to perform writes in order to continue doing reads.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-fs tests

[Smart Weblications GmbH - Florian Wiessner]

Am 05.09.2012 18:52, schrieb Gregory Farnum:
> On Wed, Sep 5, 2012 at 9:42 AM, Smart Weblications GmbH - Florian
> Wiessner <f.wiessner@smart-weblications.de> wrote:
>> Am 05.09.2012 18:22, schrieb Tommi Virtanen:
>>> On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian
>>> Wiessner <f.wiessner@smart-weblications.de> wrote:
>>>> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs.
>>>>
>>>> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few
>>>> times, then stopping one osd (osd.1), afer a while of recovering, suddenly:
>>
>>>
>>> Please provide English error messages when you share things with the
>>> list. In this case I can figure out what the message is, but really,
>>> we're all pattern matching animals and the specific strings in
>>> /usr/include/asm-generic/errno.h are what we know.
>>>
>>
>> OK, will change locales.
>>
>>>> no space left on device, but:
>>>>
>>>> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean,
>>>> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail;
>>>> 36677/215076 degraded (17.053%)
>>>>
>>>> there is space left?
>>>
>>> Only 844 MB available, with the pseudo-random placement policies,
>>> means you practically are out of space.
>>>
>>> It looks like you had only 15GB to begin with, and with typical
>>> replication, that's <5GB usable space. That is dangerously small for
>>> any real use; Ceph currently does not cope very well with running out
>>> of space.
>>>
>>
>> It is a test-cluster running on my thinkpad, its main purpose is to test cephfs,
>> there is no need for real space. I added osd.1 again, then after recovery the
>> problem went away. I forced this situation to check how cephfs will behave when
>> cluster is near-full, osd fails and ceph tries to recover until backfill fills
>> up other osds so ceph is full.
>>
>> I observed on the client that no IO was possible anymore so that the client was
>> unusable.
>>
>> Is there a smarter way to handle this? It is bad that cephfs then stalls, it
>> would be better if it just returns that there is no space left, but still allow
>> read access... can this be tuned somewhere?
> 
> What client were you using? I believe it does allow reads while full —
> but your client can pretty easily get itself into a situation where it
> needs to perform writes in order to continue doing reads.
> 

ceph-fuse argonaut 0.48.1

ls, mount, df -h etc all hanged, i had to reboot the client...

-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html