ceph kernel client orientation

ceph kernel client orientation

[Michal.Plsek]

Hello,

I am trying to get to functions responsible for reading/writing to/opening RBD blocks in ceph client kernel module (alternatives to librbd’s rbd_read(), rbd_write() etc.). I presume it should be located somewhere around drivers/block/, but until now I’ve been without luck. My idea is to edit these functions, rebuild the ceph kernel ‘rbd’ module and replace it. Since comments are pretty much missing everywhere, it would be nice to narrow my searching area.

If you know anything about it, please let me know. Thanks, M.

Re: ceph kernel client orientation

[Ilya Dryomov]

On Tue, May 19, 2020 at 3:44 PM <Michal.Plsek@seznam.cz> wrote:
>
> Hello,
>
> I am trying to get to functions responsible for reading/writing to/opening RBD blocks in ceph client kernel module (alternatives to librbd’s rbd_read(), rbd_write() etc.). I presume it should be located somewhere around drivers/block/, but until now I’ve been without luck. My idea is to edit these functions, rebuild the ceph kernel ‘rbd’ module and replace it. Since comments are pretty much missing everywhere, it would be nice to narrow my searching area.
>
> If you know anything about it, please let me know. Thanks, M.

Hi Michal,

Everything is in drivers/block/rbd.c.  The entry point is
rbd_queue_rq(), this is where all rbd requests are dispatched from.
After setting up where data is to be written from (for writes) or read
to (for reads), the details specific to each type of request (read,
write, discard or zeroout) are handled in __rbd_img_fill_request()
and then later on the respective state machine is kicked off.

The job of the state machine is to submit requests to the OSDs and
handle replies from the OSDs.  As in librbd, satisfying a single
user I/O request can require sending multiple OSDs requests, in some
cases sequentially.

Unfortunately, there is no one function to edit.  I might be able
to help more if you explain what you are trying to achieve.

Thanks,

                Ilya

Re: ceph kernel client orientation

[Michal.Plsek]

Thanks for swift answer.

(This is my usage in librbd.cc)

Basically there is a folder with symmetric keys used for block encryption, one key for one disk in some pool. For identification of key I need (pool_id, disk_id) of block. I am temporarily saving key to librbd::ImageCtx structure, so I don't have to get it from file every time. I use this key to encrypt/decrypt block data. Encrypt/decrypt is primitive, I'm not gonna mention it here, but it is done over the data provided by functions rbd_read() and rbd_write().

If you could point how to edit rbd.c content to achieve similar behaviour, I would be much obliged.

M.



---------- Původní e-mail ----------

Od: Ilya Dryomov <idryomov@gmail.com>

Komu: Michal.Plsek@seznam.cz

Datum: 19. 5. 2020 16:36:05

Předmět: Re: ceph kernel client orientation

On Tue, May 19, 2020 at 3:44 PM <Michal.Plsek@seznam.cz> wrote:

>

> Hello,

>

> I am trying to get to functions responsible for reading/writing to/opening RBD blocks in ceph client kernel module (alternatives to librbd’s rbd_read(), rbd_write() etc.). I presume it should be located somewhere around drivers/block/, but until now I’ve been without luck. My idea is to edit these functions, rebuild the ceph kernel ‘rbd’ module and replace it. Since comments are pretty much missing everywhere, it would be nice to narrow my searching area.

>

> If you know anything about it, please let me know. Thanks, M.



Hi Michal,



Everything is in drivers/block/rbd.c.  The entry point is

rbd_queue_rq(), this is where all rbd requests are dispatched from.

After setting up where data is to be written from (for writes) or read

to (for reads), the details specific to each type of request (read,

write, discard or zeroout) are handled in __rbd_img_fill_request()

and then later on the respective state machine is kicked off.



The job of the state machine is to submit requests to the OSDs and

handle replies from the OSDs.  As in librbd, satisfying a single

user I/O request can require sending multiple OSDs requests, in some

cases sequentially.



Unfortunately, there is no one function to edit.  I might be able

to help more if you explain what you are trying to achieve.



Thanks,



                Ilya

Re: ceph kernel client orientation

[Ilya Dryomov]

On Wed, May 20, 2020 at 9:36 AM <Michal.Plsek@seznam.cz> wrote:
>
> Thanks for swift answer.
>
> (This is my usage in librbd.cc)
>
> Basically there is a folder with symmetric keys used for block encryption, one key for one disk in some pool. For identification of key I need (pool_id, disk_id) of block. I am temporarily saving key to librbd::ImageCtx structure, so I don't have to get it from file every time. I use this key to encrypt/decrypt block data. Encrypt/decrypt is primitive, I'm not gonna mention it here, but it is done over the data provided by functions rbd_read() and rbd_write().
>
> If you could point how to edit rbd.c content to achieve similar behaviour, I would be much obliged.

I'm not sure what exactly you mean by disk id, but I assume image
id (displayed by "rbd info" in block_name_prefix field) is probably
part of that.  It is looked up in rbd_dev_image_id(), called from
rbd_dev_image_probe().  More generally, do_rbd_add() is roughly
equivalent to rbd_open() in librbd.  Everything related to "opening"
the image is done in or under do_rbd_add().

struct rbd_device is passed pretty much everywhere, so if you are
storing a key in librbd::ImageCtx, struct rbd_device is probably
the place to put it.

As for encryption, the easiest would probably be to stick it into
__rbd_img_fill_request().  But I want to stress that bolting on
your own crypto is very error-prone and highly unlikely to produce
anything remotely secure.  Unless you are doing it to get familiar
with the codebase or just for fun, I would advise against it.

Thanks,

                Ilya

Re: ceph kernel client orientation

[Michal.Plsek]

My mistake, I meant image id. Yesterday I decided to put en/decryption into rbd_queue_rq() and process data while traversing buffer_head structures. Do not worry, I am very much aware of Never Roll Out Your Own Crypto :), this is not my idea though. I am just working on someone else's idea.

Thanks for your help.

>

> Thanks for swift answer.

>

> (This is my usage in librbd.cc)

>

> Basically there is a folder with symmetric keys used for block encryption, one key for one disk in some pool. For identification of key I need (pool_id, disk_id) of block. I am temporarily saving key to librbd::ImageCtx structure, so I don't have to get it from file every time. I use this key to encrypt/decrypt block data. Encrypt/decrypt is primitive, I'm not gonna mention it here, but it is done over the data provided by functions rbd_read() and rbd_write().

>

> If you could point how to edit rbd.c content to achieve similar behaviour, I would be much obliged.



I'm not sure what exactly you mean by disk id, but I assume image

id (displayed by "rbd info" in block_name_prefix field) is probably

part of that.  It is looked up in rbd_dev_image_id(), called from

rbd_dev_image_probe().  More generally, do_rbd_add() is roughly

equivalent to rbd_open() in librbd.  Everything related to "opening"

the image is done in or under do_rbd_add().



struct rbd_device is passed pretty much everywhere, so if you are

storing a key in librbd::ImageCtx, struct rbd_device is probably

the place to put it.



As for encryption, the easiest would probably be to stick it into

__rbd_img_fill_request().  But I want to stress that bolting on

your own crypto is very error-prone and highly unlikely to produce

anything remotely secure.  Unless you are doing it to get familiar

with the codebase or just for fun, I would advise against it.



Thanks,



                Ilya