How to leverage ceph udev rules for containerized ceph

How to leverage ceph udev rules for containerized ceph

[Jim Curtis]

Greeting ceph-devel,

We are working on a ceph containerization project within Red Hat.  We
have recently released our RHEL-based ceph container docker image and
now we are moving on to handling a feature limitation with that image.

Specifically, the issue is that on our Atomic host, there is no ceph
installed, so there are no ceph udev rules to trigger dynamic
configuration of OSDs when a disk is plugged into the host.

What we would like to do is install our own set of ceph udev rules
that would trigger the startup of our ceph docker container.  We would
like to leverage the current implementation of the ceph udev rules to
do this.

Also, since ceph-disk and Ceph's udev rules are tightly coupled and
ceph-disk creates systemd or upstart rules for OSD daemons, does it
make sense to add hooks in ceph-disk to start up a containerized OSD
daemons either in systemd or upstart?

Can somebody in this community help us with this?

Thanks,

Jim C.

Re: How to leverage ceph udev rules for containerized ceph

[Loic Dachary]

Hi Jim,

On 14/03/2016 19:34, Jim Curtis wrote:
> Greeting ceph-devel,
> 
> We are working on a ceph containerization project within Red Hat.  We
> have recently released our RHEL-based ceph container docker image and
> now we are moving on to handling a feature limitation with that image.
> 
> Specifically, the issue is that on our Atomic host, there is no ceph
> installed, so there are no ceph udev rules to trigger dynamic
> configuration of OSDs when a disk is plugged into the host.

It would be convenient to have a standalone ceph-disk package (native or pypi) that includes udev rules and init scripts. 

> What we would like to do is install our own set of ceph udev rules
> that would trigger the startup of our ceph docker container.  We would
> like to leverage the current implementation of the ceph udev rules to
> do this.

Let say we allocate a new partition type for each existing partition type[1] so that ceph-disk knows it should docker run ceph-osd instead of running the ceph-osd daemon itself. If docker run ceph-osd has the same set of arguments as the ceph-osd daemon, the only thing to adapt is how the init system handles a container with a name instead of a daemon with a pid. Alternatively, ceph-disk could be instructed to delegate running the ceph-osd daemon to docker instead of using an init system. The later would make more sense to me because the semantic of an init system is not perfectly aligned with the docker run / stop semantic.

What I'm not sure about is how a ceph-osd running within a container can be instructed to use a given device. The only way I know is to expose /dev with --privileged which is probably too much. Is --device=/dev/sdb:/dev/sdb:rwm enough ? Is it possible, in case we only need to grant access to a partition used for journaling, to --device=/dev/sdc1:/dev/sdc1 ? 

> Also, since ceph-disk and Ceph's udev rules are tightly coupled and
> ceph-disk creates systemd or upstart rules for OSD daemons, does it
> make sense to add hooks in ceph-disk to start up a containerized OSD
> daemons either in systemd or upstart?

Yes, but that would probably not be my first choice.

Could you please provide the URL to the RHEL-based ceph container docker image you released recently ?

Cheers

> 
> Can somebody in this community help us with this?
> 
> Thanks,
> 
> Jim C.

[1] https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules#L4 etc.

> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to leverage ceph udev rules for containerized ceph

[Jim Curtis]

Hi Loic,

The image is on the registry:
https://access.redhat.com/search/#/container-images?q=ceph&p=1&sort=relevant&rows=12&srch=any&documentKind=ImageRepository

It can be pulled from:

registry.access.redhat.com/rhceph/rhceph-1.3-rhel7

Huamin might also have some input on your comments.  I forgot to cc
him on my original message and I'm not sure if he is subscribed to the
list.

Jim C.

On Mon, Mar 14, 2016 at 1:04 PM, Loic Dachary <loic@dachary.org> wrote:
> Hi Jim,
>
> On 14/03/2016 19:34, Jim Curtis wrote:
>> Greeting ceph-devel,
>>
>> We are working on a ceph containerization project within Red Hat.  We
>> have recently released our RHEL-based ceph container docker image and
>> now we are moving on to handling a feature limitation with that image.
>>
>> Specifically, the issue is that on our Atomic host, there is no ceph
>> installed, so there are no ceph udev rules to trigger dynamic
>> configuration of OSDs when a disk is plugged into the host.
>
> It would be convenient to have a standalone ceph-disk package (native or pypi) that includes udev rules and init scripts.
>
>> What we would like to do is install our own set of ceph udev rules
>> that would trigger the startup of our ceph docker container.  We would
>> like to leverage the current implementation of the ceph udev rules to
>> do this.
>
> Let say we allocate a new partition type for each existing partition type[1] so that ceph-disk knows it should docker run ceph-osd instead of running the ceph-osd daemon itself. If docker run ceph-osd has the same set of arguments as the ceph-osd daemon, the only thing to adapt is how the init system handles a container with a name instead of a daemon with a pid. Alternatively, ceph-disk could be instructed to delegate running the ceph-osd daemon to docker instead of using an init system. The later would make more sense to me because the semantic of an init system is not perfectly aligned with the docker run / stop semantic.
>
> What I'm not sure about is how a ceph-osd running within a container can be instructed to use a given device. The only way I know is to expose /dev with --privileged which is probably too much. Is --device=/dev/sdb:/dev/sdb:rwm enough ? Is it possible, in case we only need to grant access to a partition used for journaling, to --device=/dev/sdc1:/dev/sdc1 ?
>
>> Also, since ceph-disk and Ceph's udev rules are tightly coupled and
>> ceph-disk creates systemd or upstart rules for OSD daemons, does it
>> make sense to add hooks in ceph-disk to start up a containerized OSD
>> daemons either in systemd or upstart?
>
> Yes, but that would probably not be my first choice.
>
> Could you please provide the URL to the RHEL-based ceph container docker image you released recently ?
>
> Cheers
>
>>
>> Can somebody in this community help us with this?
>>
>> Thanks,
>>
>> Jim C.
>
> [1] https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules#L4 etc.
>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to leverage ceph udev rules for containerized ceph

[Loic Dachary]

On 14/03/2016 21:28, Jim Curtis wrote:
> Hi Loic,
> 
> The image is on the registry:
> https://access.redhat.com/search/#/container-images?q=ceph&p=1&sort=relevant&rows=12&srch=any&documentKind=ImageRepository
> 
> It can be pulled from:
> 
> registry.access.redhat.com/rhceph/rhceph-1.3-rhel7

Thanks ! I don't know how to retrieve the Dockerfile that was used to create this image from the registry, is there a command for this ?

> 
> Huamin might also have some input on your comments.  I forgot to cc
> him on my original message and I'm not sure if he is subscribed to the
> list.
> 
> Jim C.
> 
> On Mon, Mar 14, 2016 at 1:04 PM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Jim,
>>
>> On 14/03/2016 19:34, Jim Curtis wrote:
>>> Greeting ceph-devel,
>>>
>>> We are working on a ceph containerization project within Red Hat.  We
>>> have recently released our RHEL-based ceph container docker image and
>>> now we are moving on to handling a feature limitation with that image.
>>>
>>> Specifically, the issue is that on our Atomic host, there is no ceph
>>> installed, so there are no ceph udev rules to trigger dynamic
>>> configuration of OSDs when a disk is plugged into the host.
>>
>> It would be convenient to have a standalone ceph-disk package (native or pypi) that includes udev rules and init scripts.
>>
>>> What we would like to do is install our own set of ceph udev rules
>>> that would trigger the startup of our ceph docker container.  We would
>>> like to leverage the current implementation of the ceph udev rules to
>>> do this.
>>
>> Let say we allocate a new partition type for each existing partition type[1] so that ceph-disk knows it should docker run ceph-osd instead of running the ceph-osd daemon itself. If docker run ceph-osd has the same set of arguments as the ceph-osd daemon, the only thing to adapt is how the init system handles a container with a name instead of a daemon with a pid. Alternatively, ceph-disk could be instructed to delegate running the ceph-osd daemon to docker instead of using an init system. The later would make more sense to me because the semantic of an init system is not perfectly aligned with the docker run / stop semantic.
>>
>> What I'm not sure about is how a ceph-osd running within a container can be instructed to use a given device. The only way I know is to expose /dev with --privileged which is probably too much. Is --device=/dev/sdb:/dev/sdb:rwm enough ? Is it possible, in case we only need to grant access to a partition used for journaling, to --device=/dev/sdc1:/dev/sdc1 ?
>>
>>> Also, since ceph-disk and Ceph's udev rules are tightly coupled and
>>> ceph-disk creates systemd or upstart rules for OSD daemons, does it
>>> make sense to add hooks in ceph-disk to start up a containerized OSD
>>> daemons either in systemd or upstart?
>>
>> Yes, but that would probably not be my first choice.
>>
>> Could you please provide the URL to the RHEL-based ceph container docker image you released recently ?
>>
>> Cheers
>>
>>>
>>> Can somebody in this community help us with this?
>>>
>>> Thanks,
>>>
>>> Jim C.
>>
>> [1] https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules#L4 etc.
>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to leverage ceph udev rules for containerized ceph

[Daniel Gryniewicz]

On Tue, Mar 15, 2016 at 2:38 AM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 14/03/2016 21:28, Jim Curtis wrote:
>> Hi Loic,
>>
>> The image is on the registry:
>> https://access.redhat.com/search/#/container-images?q=ceph&p=1&sort=relevant&rows=12&srch=any&documentKind=ImageRepository
>>
>> It can be pulled from:
>>
>> registry.access.redhat.com/rhceph/rhceph-1.3-rhel7
>
> Thanks ! I don't know how to retrieve the Dockerfile that was used to create this image from the registry, is there a command for this ?

It was built from the docker branch of github.com/ceph/ceph-docker

>
>>
>> Huamin might also have some input on your comments.  I forgot to cc
>> him on my original message and I'm not sure if he is subscribed to the
>> list.
>>
>> Jim C.
>>
>> On Mon, Mar 14, 2016 at 1:04 PM, Loic Dachary <loic@dachary.org> wrote:
>>> Hi Jim,
>>>
>>> On 14/03/2016 19:34, Jim Curtis wrote:
>>>> Greeting ceph-devel,
>>>>
>>>> We are working on a ceph containerization project within Red Hat.  We
>>>> have recently released our RHEL-based ceph container docker image and
>>>> now we are moving on to handling a feature limitation with that image.
>>>>
>>>> Specifically, the issue is that on our Atomic host, there is no ceph
>>>> installed, so there are no ceph udev rules to trigger dynamic
>>>> configuration of OSDs when a disk is plugged into the host.
>>>
>>> It would be convenient to have a standalone ceph-disk package (native or pypi) that includes udev rules and init scripts.
>>>
>>>> What we would like to do is install our own set of ceph udev rules
>>>> that would trigger the startup of our ceph docker container.  We would
>>>> like to leverage the current implementation of the ceph udev rules to
>>>> do this.
>>>
>>> Let say we allocate a new partition type for each existing partition type[1] so that ceph-disk knows it should docker run ceph-osd instead of running the ceph-osd daemon itself. If docker run ceph-osd has the same set of arguments as the ceph-osd daemon, the only thing to adapt is how the init system handles a container with a name instead of a daemon with a pid. Alternatively, ceph-disk could be instructed to delegate running the ceph-osd daemon to docker instead of using an init system. The later would make more sense to me because the semantic of an init system is not perfectly aligned with the docker run / stop semantic.
>>>
>>> What I'm not sure about is how a ceph-osd running within a container can be instructed to use a given device. The only way I know is to expose /dev with --privileged which is probably too much. Is --device=/dev/sdb:/dev/sdb:rwm enough ? Is it possible, in case we only need to grant access to a partition used for journaling, to --device=/dev/sdc1:/dev/sdc1 ?

We're currently running it privileged and mounting in /dev.  The main
reason for mounting in /dev is to be able to create partitions.
Provisioning and activating are separate steps, so could be done
differently, but they currently operate the same way in our tech
preview.

>>>
>>>> Also, since ceph-disk and Ceph's udev rules are tightly coupled and
>>>> ceph-disk creates systemd or upstart rules for OSD daemons, does it
>>>> make sense to add hooks in ceph-disk to start up a containerized OSD
>>>> daemons either in systemd or upstart?
>>>
>>> Yes, but that would probably not be my first choice.
>>>
>>> Could you please provide the URL to the RHEL-based ceph container docker image you released recently ?
>>>
>>> Cheers
>>>
>>>>
>>>> Can somebody in this community help us with this?
>>>>
>>>> Thanks,
>>>>
>>>> Jim C.
>>>
>>> [1] https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules#L4 etc.
>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>> --
>>> Loïc Dachary, Artisan Logiciel Libre
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> Loïc Dachary, Artisan Logiciel Libre
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Daniel
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to leverage ceph udev rules for containerized ceph

[Loic Dachary]

On 16/03/2016 13:27, Daniel Gryniewicz wrote:
> On Tue, Mar 15, 2016 at 2:38 AM, Loic Dachary <loic@dachary.org> wrote:
>>
>>
>> On 14/03/2016 21:28, Jim Curtis wrote:
>>> Hi Loic,
>>>
>>> The image is on the registry:
>>> https://access.redhat.com/search/#/container-images?q=ceph&p=1&sort=relevant&rows=12&srch=any&documentKind=ImageRepository
>>>
>>> It can be pulled from:
>>>
>>> registry.access.redhat.com/rhceph/rhceph-1.3-rhel7
>>
>> Thanks ! I don't know how to retrieve the Dockerfile that was used to create this image from the registry, is there a command for this ?
> 
> It was built from the docker branch of github.com/ceph/ceph-docker
> 
>>
>>>
>>> Huamin might also have some input on your comments.  I forgot to cc
>>> him on my original message and I'm not sure if he is subscribed to the
>>> list.
>>>
>>> Jim C.
>>>
>>> On Mon, Mar 14, 2016 at 1:04 PM, Loic Dachary <loic@dachary.org> wrote:
>>>> Hi Jim,
>>>>
>>>> On 14/03/2016 19:34, Jim Curtis wrote:
>>>>> Greeting ceph-devel,
>>>>>
>>>>> We are working on a ceph containerization project within Red Hat.  We
>>>>> have recently released our RHEL-based ceph container docker image and
>>>>> now we are moving on to handling a feature limitation with that image.
>>>>>
>>>>> Specifically, the issue is that on our Atomic host, there is no ceph
>>>>> installed, so there are no ceph udev rules to trigger dynamic
>>>>> configuration of OSDs when a disk is plugged into the host.
>>>>
>>>> It would be convenient to have a standalone ceph-disk package (native or pypi) that includes udev rules and init scripts.
>>>>
>>>>> What we would like to do is install our own set of ceph udev rules
>>>>> that would trigger the startup of our ceph docker container.  We would
>>>>> like to leverage the current implementation of the ceph udev rules to
>>>>> do this.
>>>>
>>>> Let say we allocate a new partition type for each existing partition type[1] so that ceph-disk knows it should docker run ceph-osd instead of running the ceph-osd daemon itself. If docker run ceph-osd has the same set of arguments as the ceph-osd daemon, the only thing to adapt is how the init system handles a container with a name instead of a daemon with a pid. Alternatively, ceph-disk could be instructed to delegate running the ceph-osd daemon to docker instead of using an init system. The later would make more sense to me because the semantic of an init system is not perfectly aligned with the docker run / stop semantic.
>>>>
>>>> What I'm not sure about is how a ceph-osd running within a container can be instructed to use a given device. The only way I know is to expose /dev with --privileged which is probably too much. Is --device=/dev/sdb:/dev/sdb:rwm enough ? Is it possible, in case we only need to grant access to a partition used for journaling, to --device=/dev/sdc1:/dev/sdc1 ?
> 
> We're currently running it privileged and mounting in /dev.  The main
> reason for mounting in /dev is to be able to create partitions.
> Provisioning and activating are separate steps, so could be done
> differently, but they currently operate the same way in our tech
> preview.

It's also a valid option with the upside of not requiring any change.

>>>>
>>>>> Also, since ceph-disk and Ceph's udev rules are tightly coupled and
>>>>> ceph-disk creates systemd or upstart rules for OSD daemons, does it
>>>>> make sense to add hooks in ceph-disk to start up a containerized OSD
>>>>> daemons either in systemd or upstart?
>>>>
>>>> Yes, but that would probably not be my first choice.
>>>>
>>>> Could you please provide the URL to the RHEL-based ceph container docker image you released recently ?
>>>>
>>>> Cheers
>>>>
>>>>>
>>>>> Can somebody in this community help us with this?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jim C.
>>>>
>>>> [1] https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules#L4 etc.
>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>
>>>> --
>>>> Loïc Dachary, Artisan Logiciel Libre
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> Daniel
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to leverage ceph udev rules for containerized ceph

[Loic Dachary]

Hi,

Assuming disks are prepared using ceph-disk from within a container with access to /dev, activating using udev rules could probably be done by prefixing all ceph-disk trigger in https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules with docker run ceph-osd ceph-disk trigger. And merely installing this udev rule file on the host.

The problem is that ceph-disk trigger relies on the init system (upstart or systemd) to asynchronously run the ceph-disk activation. It does that so the udev action does not last longer than strictly necessary. If systemd can run within the container, even if it does not have access to system services, it would be enough for the purpose of ceph-disk trigger.

Another approach would be to rely on "docker run" to run ceph-disk activate in the background and make sure the udev action is short lived. That would be defeated if the ceph-osd docker image did not exist on the machine as docker run would have to download it, which can take a long time. However, it probably is safe to assume that a machine used to ceph-disk prepare disks via the ceph-osd container already has such an image. Otherwise it would not have disks to activate anyway. The udev actions could then be run with ceph-disk trigger --sync (which does *not* delegate to the init system).

Note that udev rules may fire actions that do not lead to a running ceph-osd. For instance, if /dev/sdb comes up with the journal while /dev/sdc has the data but is not there yet. The journal activation will fail and the container that tried to activate must be removed. When /dev/sdc comes up, it will try to activate and succeed because /dev/sdb is there already and the journal can be found. The container that activated successfully must keep running.

Cheers

On 14/03/2016 19:34, Jim Curtis wrote:
> Greeting ceph-devel,
> 
> We are working on a ceph containerization project within Red Hat.  We
> have recently released our RHEL-based ceph container docker image and
> now we are moving on to handling a feature limitation with that image.
> 
> Specifically, the issue is that on our Atomic host, there is no ceph
> installed, so there are no ceph udev rules to trigger dynamic
> configuration of OSDs when a disk is plugged into the host.
> 
> What we would like to do is install our own set of ceph udev rules
> that would trigger the startup of our ceph docker container.  We would
> like to leverage the current implementation of the ceph udev rules to
> do this.
> 
> Also, since ceph-disk and Ceph's udev rules are tightly coupled and
> ceph-disk creates systemd or upstart rules for OSD daemons, does it
> make sense to add hooks in ceph-disk to start up a containerized OSD
> daemons either in systemd or upstart?
> 
> Can somebody in this community help us with this?
> 
> Thanks,
> 
> Jim C.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html