* ceph-fs tests @ 2012-09-04 23:26 Smart Weblications GmbH - Florian Wiessner 2012-09-05 4:31 ` rbd 0.48 storage support for kvm proxmox distribution available Alexandre DERUMIER 2012-09-05 16:22 ` ceph-fs tests Tommi Virtanen 0 siblings, 2 replies; 15+ messages in thread From: Smart Weblications GmbH - Florian Wiessner @ 2012-09-04 23:26 UTC (permalink / raw) To: ceph-devel hi, i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs. i mount ceph via fuse, then i downloaded kernel tree and decompressed a few times, then stopping one osd (osd.1), afer a while of recovering, suddenly: tar: linux-3.5.3/drivers/media/video/zoran/zr36060.h: Kann write nicht ausführen: Auf dem Gerät ist kein Speicherplatz mehr verfügbar linux-3.5.3/drivers/media/video/zr364xx.c tar: linux-3.5.3/drivers/media/video/zr364xx.c: Kann write nicht ausführen: Auf dem Gerät ist kein Speicherplatz mehr verfügbar linux-3.5.3/drivers/memory/ no space left on device, but: 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean, 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail; 36677/215076 degraded (17.053%) there is space left? -- Mit freundlichen Grüßen, Florian Wiessner Smart Weblications GmbH Martinsberger Str. 1 D-95119 Naila fon.: +49 9282 9638 200 fax.: +49 9282 9638 205 24/7: +49 900 144 000 00 - 0,99 EUR/Min* http://www.smart-weblications.de -- Sitz der Gesellschaft: Naila Geschäftsführer: Florian Wiessner HRB-Nr.: HRB 3840 Amtsgericht Hof *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* rbd 0.48 storage support for kvm proxmox distribution available 2012-09-04 23:26 ceph-fs tests Smart Weblications GmbH - Florian Wiessner @ 2012-09-05 4:31 ` Alexandre DERUMIER 2012-09-05 9:11 ` Wido den Hollander 2012-09-05 16:22 ` ceph-fs tests Tommi Virtanen 1 sibling, 1 reply; 15+ messages in thread From: Alexandre DERUMIER @ 2012-09-05 4:31 UTC (permalink / raw) To: ceph-devel Hi List, We have added rbd 0.48 support to the proxmox 2.1 kvm distribution http://www.proxmox.com/products/proxmox-ve Proxmox setup: edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) rbd: mycephcluster monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 pool rbd username admin authsupported cephx;none content images then you need to copy the keyring file from ceph to proxmox scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring For now, you can add/delete/resize rbd volumes from gui. Snapshots/cloning will be added soon (when layering will be available) Regards, Alexandre Derumier ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 4:31 ` rbd 0.48 storage support for kvm proxmox distribution available Alexandre DERUMIER @ 2012-09-05 9:11 ` Wido den Hollander 2012-09-05 9:30 ` Alexandre DERUMIER 0 siblings, 1 reply; 15+ messages in thread From: Wido den Hollander @ 2012-09-05 9:11 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: ceph-devel On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote: > Hi List, > > We have added rbd 0.48 support to the proxmox 2.1 kvm distribution > http://www.proxmox.com/products/proxmox-ve > > > Proxmox setup: > > edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) > > rbd: mycephcluster > monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 > pool rbd > username admin > authsupported cephx;none > content images > Proxmox doesn't use libvirt, does it? Any plans to implement the RBD caching? Nice work though! Wido > > then you need to copy the keyring file from ceph to proxmox > > scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring > > > > For now, you can add/delete/resize rbd volumes from gui. > Snapshots/cloning will be added soon (when layering will be available) > > > Regards, > > Alexandre Derumier > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 9:11 ` Wido den Hollander @ 2012-09-05 9:30 ` Alexandre DERUMIER 2012-09-05 12:35 ` Wido den Hollander 0 siblings, 1 reply; 15+ messages in thread From: Alexandre DERUMIER @ 2012-09-05 9:30 UTC (permalink / raw) To: Wido den Hollander; +Cc: ceph-devel >>Proxmox doesn't use libvirt, does it? Yes, we don't use libvirt. >>Any plans to implement the RBD caching? It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days) Tunning of cache size can be done with a /etc/ceph.conf file ----- Mail original ----- De: "Wido den Hollander" <wido@widodh.nl> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: ceph-devel@vger.kernel.org Envoyé: Mercredi 5 Septembre 2012 11:11:27 Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote: > Hi List, > > We have added rbd 0.48 support to the proxmox 2.1 kvm distribution > http://www.proxmox.com/products/proxmox-ve > > > Proxmox setup: > > edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) > > rbd: mycephcluster > monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 > pool rbd > username admin > authsupported cephx;none > content images > Proxmox doesn't use libvirt, does it? Any plans to implement the RBD caching? Nice work though! Wido > > then you need to copy the keyring file from ceph to proxmox > > scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring > > > > For now, you can add/delete/resize rbd volumes from gui. > Snapshots/cloning will be added soon (when layering will be available) > > > Regards, > > Alexandre Derumier > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 9:30 ` Alexandre DERUMIER @ 2012-09-05 12:35 ` Wido den Hollander 2012-09-05 14:00 ` Alexandre DERUMIER ` (2 more replies) 0 siblings, 3 replies; 15+ messages in thread From: Wido den Hollander @ 2012-09-05 12:35 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: ceph-devel On 09/05/2012 11:30 AM, Alexandre DERUMIER wrote: >>> Proxmox doesn't use libvirt, does it? > Yes, we don't use libvirt. > >>> Any plans to implement the RBD caching? > > It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days) > > Tunning of cache size can be done with a /etc/ceph.conf file > That is kind of dangerous imho and for a couple of reasons. For configuring the storage you have /etc/pve/storage.cfg where you can add the RBD pool, configure the monitors and cephx, but for caching you rely in librbd reading ceph.conf? That is hidden from the user, reading /etc/ceph/ceph.conf will go without your knowledge. I'd opt for passing down all the options to Qemu and being able to run without a ceph.conf I've ran into the same problems with libvirt and CloudStack. I couldn't figure out why libvirt was still able to connect to a specific cluster until I found out my ceph.conf was still in place. I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by default with librbd/librados to take away these kind of issues. And you would also have this weird situation where the ceph.conf could have a couple of monitor entries and your "storage.cfg", how will that work out? I would try not to rely on the ceph.conf at all and have Proxmox pass all the configuration options down to Qemu. Wido > > > ----- Mail original ----- > > De: "Wido den Hollander" <wido@widodh.nl> > À: "Alexandre DERUMIER" <aderumier@odiso.com> > Cc: ceph-devel@vger.kernel.org > Envoyé: Mercredi 5 Septembre 2012 11:11:27 > Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available > > On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote: >> Hi List, >> >> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution >> http://www.proxmox.com/products/proxmox-ve >> >> >> Proxmox setup: >> >> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) >> >> rbd: mycephcluster >> monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 >> pool rbd >> username admin >> authsupported cephx;none >> content images >> > > Proxmox doesn't use libvirt, does it? > > Any plans to implement the RBD caching? > > Nice work though! > > Wido > >> >> then you need to copy the keyring file from ceph to proxmox >> >> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring >> >> >> >> For now, you can add/delete/resize rbd volumes from gui. >> Snapshots/cloning will be added soon (when layering will be available) >> >> >> Regards, >> >> Alexandre Derumier >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 12:35 ` Wido den Hollander @ 2012-09-05 14:00 ` Alexandre DERUMIER 2012-09-05 14:47 ` Josh Durgin 2012-09-05 16:40 ` Tommi Virtanen 2 siblings, 0 replies; 15+ messages in thread From: Alexandre DERUMIER @ 2012-09-05 14:00 UTC (permalink / raw) To: Wido den Hollander; +Cc: ceph-devel Thanks for the infos. I dind't have documented the ceph.conf feature,it's more a workaround for now for users which want to tune some values. Indeed,I'm planning to add the cache size options to the proxmox storage.cfg. Thanks Alexandre ----- Mail original ----- De: "Wido den Hollander" <wido@widodh.nl> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: ceph-devel@vger.kernel.org Envoyé: Mercredi 5 Septembre 2012 14:35:16 Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available On 09/05/2012 11:30 AM, Alexandre DERUMIER wrote: >>> Proxmox doesn't use libvirt, does it? > Yes, we don't use libvirt. > >>> Any plans to implement the RBD caching? > > It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days) > > Tunning of cache size can be done with a /etc/ceph.conf file > That is kind of dangerous imho and for a couple of reasons. For configuring the storage you have /etc/pve/storage.cfg where you can add the RBD pool, configure the monitors and cephx, but for caching you rely in librbd reading ceph.conf? That is hidden from the user, reading /etc/ceph/ceph.conf will go without your knowledge. I'd opt for passing down all the options to Qemu and being able to run without a ceph.conf I've ran into the same problems with libvirt and CloudStack. I couldn't figure out why libvirt was still able to connect to a specific cluster until I found out my ceph.conf was still in place. I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by default with librbd/librados to take away these kind of issues. And you would also have this weird situation where the ceph.conf could have a couple of monitor entries and your "storage.cfg", how will that work out? I would try not to rely on the ceph.conf at all and have Proxmox pass all the configuration options down to Qemu. Wido > > > ----- Mail original ----- > > De: "Wido den Hollander" <wido@widodh.nl> > À: "Alexandre DERUMIER" <aderumier@odiso.com> > Cc: ceph-devel@vger.kernel.org > Envoyé: Mercredi 5 Septembre 2012 11:11:27 > Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available > > On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote: >> Hi List, >> >> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution >> http://www.proxmox.com/products/proxmox-ve >> >> >> Proxmox setup: >> >> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) >> >> rbd: mycephcluster >> monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 >> pool rbd >> username admin >> authsupported cephx;none >> content images >> > > Proxmox doesn't use libvirt, does it? > > Any plans to implement the RBD caching? > > Nice work though! > > Wido > >> >> then you need to copy the keyring file from ceph to proxmox >> >> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring >> >> >> >> For now, you can add/delete/resize rbd volumes from gui. >> Snapshots/cloning will be added soon (when layering will be available) >> >> >> Regards, >> >> Alexandre Derumier >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 12:35 ` Wido den Hollander 2012-09-05 14:00 ` Alexandre DERUMIER @ 2012-09-05 14:47 ` Josh Durgin 2012-09-05 16:40 ` Tommi Virtanen 2 siblings, 0 replies; 15+ messages in thread From: Josh Durgin @ 2012-09-05 14:47 UTC (permalink / raw) To: Wido den Hollander; +Cc: Alexandre DERUMIER, ceph-devel On Wed, 05 Sep 2012 14:35:16 +0200, Wido den Hollander <wido@widodh.nl> wrote: > On 09/05/2012 11:30 AM, Alexandre DERUMIER wrote: >>>> Proxmox doesn't use libvirt, does it? >> Yes, we don't use libvirt. >> >>>> Any plans to implement the RBD caching? >> >> It's already implemented (cache=writeback), patched qemu-kvm 1.1. (and qemu-kvm 1.2 is coming in the next days) >> >> Tunning of cache size can be done with a /etc/ceph.conf file >> > > That is kind of dangerous imho and for a couple of reasons. > > For configuring the storage you have /etc/pve/storage.cfg where you > can add the RBD pool, configure the monitors and cephx, but for > caching you rely in librbd reading ceph.conf? > > That is hidden from the user, reading /etc/ceph/ceph.conf will go > without your knowledge. I'd opt for passing down all the options to > Qemu and being able to run without a ceph.conf > > I've ran into the same problems with libvirt and CloudStack. I > couldn't figure out why libvirt was still able to connect to a > specific cluster until I found out my ceph.conf was still in place. > > I also thought it is on the roadmap to not read /etc/ceph/ceph.conf > by default with librbd/librados to take away these kind of issues. I don't think we'll want to change the default behavior (qemu reading /etc/ceph/ceph.conf) for backwards compatibility, but I agree that we should avoid relying on it in the future. Josh > And you would also have this weird situation where the ceph.conf > could have a couple of monitor entries and your "storage.cfg", how > will that work out? > > I would try not to rely on the ceph.conf at all and have Proxmox pass > all the configuration options down to Qemu. > > Wido > >> >> >> ----- Mail original ----- >> >> De: "Wido den Hollander" <wido@widodh.nl> >> À: "Alexandre DERUMIER" <aderumier@odiso.com> >> Cc: ceph-devel@vger.kernel.org >> Envoyé: Mercredi 5 Septembre 2012 11:11:27 >> Objet: Re: rbd 0.48 storage support for kvm proxmox distribution available >> >> On 09/05/2012 06:31 AM, Alexandre DERUMIER wrote: >>> Hi List, >>> >>> We have added rbd 0.48 support to the proxmox 2.1 kvm distribution >>> http://www.proxmox.com/products/proxmox-ve >>> >>> >>> Proxmox setup: >>> >>> edit the /etc/pve/storage.cfg and add the configuration (gui creation is not available yet) >>> >>> rbd: mycephcluster >>> monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789 >>> pool rbd >>> username admin >>> authsupported cephx;none >>> content images >>> >> >> Proxmox doesn't use libvirt, does it? >> >> Any plans to implement the RBD caching? >> >> Nice work though! >> >> Wido >> >>> >>> then you need to copy the keyring file from ceph to proxmox >>> >>> scp cephserver1:/etc/ceph/client.admin.keyring /etc/pve/priv/ceph/mycephcluster.keyring >>> >>> >>> >>> For now, you can add/delete/resize rbd volumes from gui. >>> Snapshots/cloning will be added soon (when layering will be available) >>> >>> >>> Regards, >>> >>> Alexandre Derumier >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 12:35 ` Wido den Hollander 2012-09-05 14:00 ` Alexandre DERUMIER 2012-09-05 14:47 ` Josh Durgin @ 2012-09-05 16:40 ` Tommi Virtanen 2012-09-05 16:43 ` Tommi Virtanen 2012-09-05 16:44 ` Josh Durgin 2 siblings, 2 replies; 15+ messages in thread From: Tommi Virtanen @ 2012-09-05 16:40 UTC (permalink / raw) To: Wido den Hollander; +Cc: Alexandre DERUMIER, ceph-devel On Wed, Sep 5, 2012 at 5:35 AM, Wido den Hollander <wido@widodh.nl> wrote: > I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by > default with librbd/librados to take away these kind of issues. Hmm. I'm not intimately familiar with librbd, but it seems it just takes RADOS ioctx as an argument, and as far as I know (/can experiment in 2 minutes) you definitely have a choice of whether librados reads a config file or not.. rbd the command line tool does read a config file by default; pass --conf=/dev/null to prevent that, then pass what ever config options you need to set as --foo-bar=baz. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 16:40 ` Tommi Virtanen @ 2012-09-05 16:43 ` Tommi Virtanen 2012-09-05 16:44 ` Josh Durgin 1 sibling, 0 replies; 15+ messages in thread From: Tommi Virtanen @ 2012-09-05 16:43 UTC (permalink / raw) To: Wido den Hollander; +Cc: Alexandre DERUMIER, ceph-devel On Wed, Sep 5, 2012 at 9:40 AM, Tommi Virtanen <tv@inktank.com> wrote: > On Wed, Sep 5, 2012 at 5:35 AM, Wido den Hollander <wido@widodh.nl> wrote: >> I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by >> default with librbd/librados to take away these kind of issues. > > Hmm. I'm not intimately familiar with librbd, but it seems it just > takes RADOS ioctx as an argument, and as far as I know (/can > experiment in 2 minutes) you definitely have a choice of whether > librados reads a config file or not.. > > rbd the command line tool does read a config file by default; pass > --conf=/dev/null to prevent that, then pass what ever config options > you need to set as --foo-bar=baz. So *QEmu* makes librados read a config file. You can avoid that just like with the rbd command line tool, pass conf=/dev/null. Quoting git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git block/rbd.c: * The "conf" option specifies a Ceph configuration file to read. If * it is not specified, we will read from the default Ceph locations * (e.g., /etc/ceph/ceph.conf). To avoid reading _any_ configuration * file, specify conf=/dev/null. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: rbd 0.48 storage support for kvm proxmox distribution available 2012-09-05 16:40 ` Tommi Virtanen 2012-09-05 16:43 ` Tommi Virtanen @ 2012-09-05 16:44 ` Josh Durgin 1 sibling, 0 replies; 15+ messages in thread From: Josh Durgin @ 2012-09-05 16:44 UTC (permalink / raw) To: Tommi Virtanen; +Cc: Wido den Hollander, Alexandre DERUMIER, ceph-devel On 09/05/2012 09:40 AM, Tommi Virtanen wrote: > On Wed, Sep 5, 2012 at 5:35 AM, Wido den Hollander <wido@widodh.nl> wrote: >> I also thought it is on the roadmap to not read /etc/ceph/ceph.conf by >> default with librbd/librados to take away these kind of issues. > > Hmm. I'm not intimately familiar with librbd, but it seems it just > takes RADOS ioctx as an argument, and as far as I know (/can > experiment in 2 minutes) you definitely have a choice of whether > librados reads a config file or not.. > > rbd the command line tool does read a config file by default; pass > --conf=/dev/null to prevent that, then pass what ever config options > you need to set as --foo-bar=baz. That's right, all the configuration is done by calls to the rados_conf_* functions [1] before calling rados_connect. It looks like the libvirt storage pool doesn't call rados_conf_read_file(cluster, NULL), so it won't read /etc/ceph/ceph.conf. Josh [1] http://ceph.com/docs/master/api/librados/#rados_conf_read_file ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: ceph-fs tests 2012-09-04 23:26 ceph-fs tests Smart Weblications GmbH - Florian Wiessner 2012-09-05 4:31 ` rbd 0.48 storage support for kvm proxmox distribution available Alexandre DERUMIER @ 2012-09-05 16:22 ` Tommi Virtanen 2012-09-05 16:37 ` Gregory Farnum 2012-09-05 16:42 ` Smart Weblications GmbH - Florian Wiessner 1 sibling, 2 replies; 15+ messages in thread From: Tommi Virtanen @ 2012-09-05 16:22 UTC (permalink / raw) To: f.wiessner; +Cc: ceph-devel On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian Wiessner <f.wiessner@smart-weblications.de> wrote: > i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs. > > i mount ceph via fuse, then i downloaded kernel tree and decompressed a few > times, then stopping one osd (osd.1), afer a while of recovering, suddenly: > > tar: linux-3.5.3/drivers/media/video/zoran/zr36060.h: Kann write nicht > ausführen: Auf dem Gerät ist kein Speicherplatz mehr verfügbar > linux-3.5.3/drivers/media/video/zr364xx.c > tar: linux-3.5.3/drivers/media/video/zr364xx.c: Kann write nicht ausführen: Auf > dem Gerät ist kein Speicherplatz mehr verfügbar > linux-3.5.3/drivers/memory/ Please provide English error messages when you share things with the list. In this case I can figure out what the message is, but really, we're all pattern matching animals and the specific strings in /usr/include/asm-generic/errno.h are what we know. > no space left on device, but: > > 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean, > 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail; > 36677/215076 degraded (17.053%) > > there is space left? Only 844 MB available, with the pseudo-random placement policies, means you practically are out of space. It looks like you had only 15GB to begin with, and with typical replication, that's <5GB usable space. That is dangerously small for any real use; Ceph currently does not cope very well with running out of space. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: ceph-fs tests 2012-09-05 16:22 ` ceph-fs tests Tommi Virtanen @ 2012-09-05 16:37 ` Gregory Farnum 2012-09-05 16:42 ` Smart Weblications GmbH - Florian Wiessner 1 sibling, 0 replies; 15+ messages in thread From: Gregory Farnum @ 2012-09-05 16:37 UTC (permalink / raw) To: Tommi Virtanen; +Cc: f.wiessner, ceph-devel On Wed, Sep 5, 2012 at 9:22 AM, Tommi Virtanen <tv@inktank.com> wrote: > On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian > Wiessner <f.wiessner@smart-weblications.de> wrote: >> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs. >> >> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few >> times, then stopping one osd (osd.1), afer a while of recovering, suddenly: >> >> tar: linux-3.5.3/drivers/media/video/zoran/zr36060.h: Kann write nicht >> ausführen: Auf dem Gerät ist kein Speicherplatz mehr verfügbar >> linux-3.5.3/drivers/media/video/zr364xx.c >> tar: linux-3.5.3/drivers/media/video/zr364xx.c: Kann write nicht ausführen: Auf >> dem Gerät ist kein Speicherplatz mehr verfügbar >> linux-3.5.3/drivers/memory/ > > Please provide English error messages when you share things with the > list. In this case I can figure out what the message is, but really, > we're all pattern matching animals and the specific strings in > /usr/include/asm-generic/errno.h are what we know. > >> no space left on device, but: >> >> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean, >> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail; >> 36677/215076 degraded (17.053%) >> >> there is space left? > > Only 844 MB available, with the pseudo-random placement policies, > means you practically are out of space. > > It looks like you had only 15GB to begin with, and with typical > replication, that's <5GB usable space. That is dangerously small for > any real use; Ceph currently does not cope very well with running out > of space. In this particular case, one of the OSDs is more than 95% full and has been marked as full (which stops cluster IO) to prevent those catastrophic failures from occurring. If you look at the full output of ceph -s, you should see a warning about having OSDs full and near-full. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: ceph-fs tests 2012-09-05 16:22 ` ceph-fs tests Tommi Virtanen 2012-09-05 16:37 ` Gregory Farnum @ 2012-09-05 16:42 ` Smart Weblications GmbH - Florian Wiessner 2012-09-05 16:52 ` Gregory Farnum 1 sibling, 1 reply; 15+ messages in thread From: Smart Weblications GmbH - Florian Wiessner @ 2012-09-05 16:42 UTC (permalink / raw) To: Tommi Virtanen, ceph-devel Am 05.09.2012 18:22, schrieb Tommi Virtanen: > On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian > Wiessner <f.wiessner@smart-weblications.de> wrote: >> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs. >> >> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few >> times, then stopping one osd (osd.1), afer a while of recovering, suddenly: > > Please provide English error messages when you share things with the > list. In this case I can figure out what the message is, but really, > we're all pattern matching animals and the specific strings in > /usr/include/asm-generic/errno.h are what we know. > OK, will change locales. >> no space left on device, but: >> >> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean, >> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail; >> 36677/215076 degraded (17.053%) >> >> there is space left? > > Only 844 MB available, with the pseudo-random placement policies, > means you practically are out of space. > > It looks like you had only 15GB to begin with, and with typical > replication, that's <5GB usable space. That is dangerously small for > any real use; Ceph currently does not cope very well with running out > of space. > It is a test-cluster running on my thinkpad, its main purpose is to test cephfs, there is no need for real space. I added osd.1 again, then after recovery the problem went away. I forced this situation to check how cephfs will behave when cluster is near-full, osd fails and ceph tries to recover until backfill fills up other osds so ceph is full. I observed on the client that no IO was possible anymore so that the client was unusable. Is there a smarter way to handle this? It is bad that cephfs then stalls, it would be better if it just returns that there is no space left, but still allow read access... can this be tuned somewhere? -- Mit freundlichen Grüßen, Florian Wiessner Smart Weblications GmbH Martinsberger Str. 1 D-95119 Naila fon.: +49 9282 9638 200 fax.: +49 9282 9638 205 24/7: +49 900 144 000 00 - 0,99 EUR/Min* http://www.smart-weblications.de -- Sitz der Gesellschaft: Naila Geschäftsführer: Florian Wiessner HRB-Nr.: HRB 3840 Amtsgericht Hof *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: ceph-fs tests 2012-09-05 16:42 ` Smart Weblications GmbH - Florian Wiessner @ 2012-09-05 16:52 ` Gregory Farnum 2012-09-06 23:22 ` Smart Weblications GmbH - Florian Wiessner 0 siblings, 1 reply; 15+ messages in thread From: Gregory Farnum @ 2012-09-05 16:52 UTC (permalink / raw) To: f.wiessner; +Cc: Tommi Virtanen, ceph-devel On Wed, Sep 5, 2012 at 9:42 AM, Smart Weblications GmbH - Florian Wiessner <f.wiessner@smart-weblications.de> wrote: > Am 05.09.2012 18:22, schrieb Tommi Virtanen: >> On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian >> Wiessner <f.wiessner@smart-weblications.de> wrote: >>> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs. >>> >>> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few >>> times, then stopping one osd (osd.1), afer a while of recovering, suddenly: > >> >> Please provide English error messages when you share things with the >> list. In this case I can figure out what the message is, but really, >> we're all pattern matching animals and the specific strings in >> /usr/include/asm-generic/errno.h are what we know. >> > > OK, will change locales. > >>> no space left on device, but: >>> >>> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean, >>> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail; >>> 36677/215076 degraded (17.053%) >>> >>> there is space left? >> >> Only 844 MB available, with the pseudo-random placement policies, >> means you practically are out of space. >> >> It looks like you had only 15GB to begin with, and with typical >> replication, that's <5GB usable space. That is dangerously small for >> any real use; Ceph currently does not cope very well with running out >> of space. >> > > It is a test-cluster running on my thinkpad, its main purpose is to test cephfs, > there is no need for real space. I added osd.1 again, then after recovery the > problem went away. I forced this situation to check how cephfs will behave when > cluster is near-full, osd fails and ceph tries to recover until backfill fills > up other osds so ceph is full. > > I observed on the client that no IO was possible anymore so that the client was > unusable. > > Is there a smarter way to handle this? It is bad that cephfs then stalls, it > would be better if it just returns that there is no space left, but still allow > read access... can this be tuned somewhere? What client were you using? I believe it does allow reads while full — but your client can pretty easily get itself into a situation where it needs to perform writes in order to continue doing reads. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: ceph-fs tests 2012-09-05 16:52 ` Gregory Farnum @ 2012-09-06 23:22 ` Smart Weblications GmbH - Florian Wiessner 0 siblings, 0 replies; 15+ messages in thread From: Smart Weblications GmbH - Florian Wiessner @ 2012-09-06 23:22 UTC (permalink / raw) To: Gregory Farnum; +Cc: Tommi Virtanen, ceph-devel Am 05.09.2012 18:52, schrieb Gregory Farnum: > On Wed, Sep 5, 2012 at 9:42 AM, Smart Weblications GmbH - Florian > Wiessner <f.wiessner@smart-weblications.de> wrote: >> Am 05.09.2012 18:22, schrieb Tommi Virtanen: >>> On Tue, Sep 4, 2012 at 4:26 PM, Smart Weblications GmbH - Florian >>> Wiessner <f.wiessner@smart-weblications.de> wrote: >>>> i set up a 3 node ceph cluster 0.48.1argonaut to test ceph-fs. >>>> >>>> i mount ceph via fuse, then i downloaded kernel tree and decompressed a few >>>> times, then stopping one osd (osd.1), afer a while of recovering, suddenly: >> >>> >>> Please provide English error messages when you share things with the >>> list. In this case I can figure out what the message is, but really, >>> we're all pattern matching animals and the specific strings in >>> /usr/include/asm-generic/errno.h are what we know. >>> >> >> OK, will change locales. >> >>>> no space left on device, but: >>>> >>>> 2012-09-04 18:46:38.242840 mon.0 [INF] pgmap v2883: 576 pgs: 512 active+clean, >>>> 64 active+recovering; 1250 MB data, 14391 MB used, 844 MB / 15236 MB avail; >>>> 36677/215076 degraded (17.053%) >>>> >>>> there is space left? >>> >>> Only 844 MB available, with the pseudo-random placement policies, >>> means you practically are out of space. >>> >>> It looks like you had only 15GB to begin with, and with typical >>> replication, that's <5GB usable space. That is dangerously small for >>> any real use; Ceph currently does not cope very well with running out >>> of space. >>> >> >> It is a test-cluster running on my thinkpad, its main purpose is to test cephfs, >> there is no need for real space. I added osd.1 again, then after recovery the >> problem went away. I forced this situation to check how cephfs will behave when >> cluster is near-full, osd fails and ceph tries to recover until backfill fills >> up other osds so ceph is full. >> >> I observed on the client that no IO was possible anymore so that the client was >> unusable. >> >> Is there a smarter way to handle this? It is bad that cephfs then stalls, it >> would be better if it just returns that there is no space left, but still allow >> read access... can this be tuned somewhere? > > What client were you using? I believe it does allow reads while full — > but your client can pretty easily get itself into a situation where it > needs to perform writes in order to continue doing reads. > ceph-fuse argonaut 0.48.1 ls, mount, df -h etc all hanged, i had to reboot the client... -- Mit freundlichen Grüßen, Florian Wiessner Smart Weblications GmbH Martinsberger Str. 1 D-95119 Naila fon.: +49 9282 9638 200 fax.: +49 9282 9638 205 24/7: +49 900 144 000 00 - 0,99 EUR/Min* http://www.smart-weblications.de -- Sitz der Gesellschaft: Naila Geschäftsführer: Florian Wiessner HRB-Nr.: HRB 3840 Amtsgericht Hof *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2012-09-06 23:23 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-09-04 23:26 ceph-fs tests Smart Weblications GmbH - Florian Wiessner 2012-09-05 4:31 ` rbd 0.48 storage support for kvm proxmox distribution available Alexandre DERUMIER 2012-09-05 9:11 ` Wido den Hollander 2012-09-05 9:30 ` Alexandre DERUMIER 2012-09-05 12:35 ` Wido den Hollander 2012-09-05 14:00 ` Alexandre DERUMIER 2012-09-05 14:47 ` Josh Durgin 2012-09-05 16:40 ` Tommi Virtanen 2012-09-05 16:43 ` Tommi Virtanen 2012-09-05 16:44 ` Josh Durgin 2012-09-05 16:22 ` ceph-fs tests Tommi Virtanen 2012-09-05 16:37 ` Gregory Farnum 2012-09-05 16:42 ` Smart Weblications GmbH - Florian Wiessner 2012-09-05 16:52 ` Gregory Farnum 2012-09-06 23:22 ` Smart Weblications GmbH - Florian Wiessner
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.