Standalone libcrush: call for participation

Standalone libcrush: call for participation

[Loic Dachary]

Hi Ceph,

The idea of packaging CRUSH [1] as a standalone library that can be used by Ceph and other projects is not new. 

From the beginning Sage expected/hoped that the algorithm would be useful for lots of different applications. Other storage systems, load balancers, CDNs, or any other distributed thing where a compact description of how to map a large number of things to other things would be useful.  This has partly been proven out in that some storage systems "use CRUSH," but they use it in the sense that they reimplemented something based on the underlying algorithm.

It's also a place where we haven't seen a need to do anything ceph-specific; the problems we've fixed and changes we've made are general issues with the original algorithm and implementation.  The main barrier has been that the code and tools are built using a lot of the generic ceph infrastructure like encode/decode/bufferlists.

The beginning of the next ceph release cycle seems like a good time to initiate that effort. If someone is willing to help with the bootstrap, now is a good time. The library will be hosted under the libcrush.org domain name. But that's about the only thing that happened so far :-)

Cheers

[1] https://github.com/ceph/ceph/tree/master/src/crush

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: Standalone libcrush: call for participation

[Adam C. Emerson]

[-- Attachment #1: Type: text/plain, Size: 744 bytes --]

On 21/01/2017, Loic Dachary wrote:
[snip]
> The beginning of the next ceph release cycle seems like a good time
> to initiate that effort. If someone is willing to help with the
> bootstrap, now is a good time. The library will be hosted under the
> libcrush.org domain name. But that's about the only thing that
> happened so far :-)

Would the Ceph git repository continue to have its own version of
CRUSH included, or would we pull in libcrush as a submodule?

Also, does this affect the relationship between userspace CRUSH and
Linux kernel CRUSH?

-- 
Senior Software Engineer           Red Hat Storage, Ann Arbor, MI, US
IRC: Aemerson@{RedHat, OFTC}
0x80F7544B90EDBFB9 E707 86BA 0C1B 62CC 152C  7C12 80F7 544B 90ED BFB9

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 603 bytes --]

Re: Standalone libcrush: call for participation

[Loic Dachary]

Hi Adam,

On 01/23/2017 05:29 PM, Adam C. Emerson wrote:
> On 21/01/2017, Loic Dachary wrote:
> [snip]
>> The beginning of the next ceph release cycle seems like a good time
>> to initiate that effort. If someone is willing to help with the
>> bootstrap, now is a good time. The library will be hosted under the
>> libcrush.org domain name. But that's about the only thing that
>> happened so far :-)
> 
> Would the Ceph git repository continue to have its own version of
> CRUSH included, or would we pull in libcrush as a submodule?

If all goes well and libcrush becomes actively and reliably maintained outside of the Ceph tree, it could either be a submodule or even a packaged dependency. Before that happens things can stay the way they are. I volunteer to manually monitor and cherry-pick whatever changes in the Ceph tree onto the standalone repository ( http://libcrush.org/main/libcrush/ ) so that it does not drift away.

> Also, does this affect the relationship between userspace CRUSH and
> Linux kernel CRUSH?

I don't know :-) Unless I'm mistaken the code in the linux kernel is an unmodified copy of the code from src/crush/*.[ch]. I however have no clue how it is tested to work as it should in both contexts. Other than running integration tests with teuthology.

How do you suggest we go about it ?

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: Standalone libcrush: call for participation

[Adam C. Emerson]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

On 23/01/2017, Loic Dachary wrote:
> Hi Adam, I don't know :-) Unless I'm mistaken the code in the linux
> kernel is an unmodified copy of the code from src/crush/*.[ch]. I
> however have no clue how it is tested to work as it should in both
> contexts. Other than running integration tests with teuthology.
> 
> How do you suggest we go about it ?

I don't actually know. I just wondered, I had the vague impression
that the CRUSH from Ceph was copied into the kernel but I didn't know
who did it, if it was parallel development or a blanket copy or
what.

It /does/ mean that any changes to libcrush have to be kept as kernel
compatible C code, which isn't bad, it's just good to make sure it's
explicit.

-- 
Senior Software Engineer           Red Hat Storage, Ann Arbor, MI, US
IRC: Aemerson@{RedHat, OFTC}
0x80F7544B90EDBFB9 E707 86BA 0C1B 62CC 152C  7C12 80F7 544B 90ED BFB9

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 603 bytes --]

Re: Standalone libcrush: call for participation

[Ilya Dryomov]

On Mon, Jan 23, 2017 at 6:00 PM, Loic Dachary <loic@dachary.org> wrote:
> Hi Adam,
>
> On 01/23/2017 05:29 PM, Adam C. Emerson wrote:
>> On 21/01/2017, Loic Dachary wrote:
>> [snip]
>>> The beginning of the next ceph release cycle seems like a good time
>>> to initiate that effort. If someone is willing to help with the
>>> bootstrap, now is a good time. The library will be hosted under the
>>> libcrush.org domain name. But that's about the only thing that
>>> happened so far :-)
>>
>> Would the Ceph git repository continue to have its own version of
>> CRUSH included, or would we pull in libcrush as a submodule?
>
> If all goes well and libcrush becomes actively and reliably maintained outside of the Ceph tree, it could either be a submodule or even a packaged dependency. Before that happens things can stay the way they are. I volunteer to manually monitor and cherry-pick whatever changes in the Ceph tree onto the standalone repository ( http://libcrush.org/main/libcrush/ ) so that it does not drift away.
>
>> Also, does this affect the relationship between userspace CRUSH and
>> Linux kernel CRUSH?
>
> I don't know :-) Unless I'm mistaken the code in the linux kernel is an unmodified copy of the code from src/crush/*.[ch].

Correct.  libcrush won't impact the kernel in any way, as long as the
C core <-> C++ wrappers boundary stays intact, which it should ;)

Some documentation will probably need to be added to cover things like
basic kernel compatibility (no floating point, stubs for asserts, etc)
and coding style for the C bits.  The core algorithm is far from where
libcrush initial activities would be centered though, I imagine.

> I however have no clue how it is tested to work as it should in both contexts. Other than running integration tests with teuthology.
>
> How do you suggest we go about it ?

I don't think we need anything special here.

Thanks,

                Ilya

Re: Standalone libcrush: call for participation

[Loic Dachary]

Hi Ilya,

On 01/23/2017 07:43 PM, Ilya Dryomov wrote:
> On Mon, Jan 23, 2017 at 6:00 PM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Adam,
>>
>> On 01/23/2017 05:29 PM, Adam C. Emerson wrote:
>>> On 21/01/2017, Loic Dachary wrote:
>>> [snip]
>>>> The beginning of the next ceph release cycle seems like a good time
>>>> to initiate that effort. If someone is willing to help with the
>>>> bootstrap, now is a good time. The library will be hosted under the
>>>> libcrush.org domain name. But that's about the only thing that
>>>> happened so far :-)
>>>
>>> Would the Ceph git repository continue to have its own version of
>>> CRUSH included, or would we pull in libcrush as a submodule?
>>
>> If all goes well and libcrush becomes actively and reliably maintained outside of the Ceph tree, it could either be a submodule or even a packaged dependency. Before that happens things can stay the way they are. I volunteer to manually monitor and cherry-pick whatever changes in the Ceph tree onto the standalone repository ( http://libcrush.org/main/libcrush/ ) so that it does not drift away.
>>
>>> Also, does this affect the relationship between userspace CRUSH and
>>> Linux kernel CRUSH?
>>
>> I don't know :-) Unless I'm mistaken the code in the linux kernel is an unmodified copy of the code from src/crush/*.[ch].
> 
> Correct.  libcrush won't impact the kernel in any way, as long as the
> C core <-> C++ wrappers boundary stays intact, which it should ;)

Yeah :-) Just rephrasing so I'm sure I understand you correctly: the important thing is that the C API stays as it is (functions & structs & constants) and that the semantic of each function is backward compatible. As long as it holds true, the kernel integration is going to be ok.

> Some documentation will probably need to be added to cover things like
> basic kernel compatibility (no floating point, stubs for asserts, etc)
> and coding style for the C bits.  

Ack.

> The core algorithm is far from where
> libcrush initial activities would be centered though, I imagine.

Indeed, more documentation & packaging right now.

>> I however have no clue how it is tested to work as it should in both contexts. Other than running integration tests with teuthology.
>>
>> How do you suggest we go about it ?
> 
> I don't think we need anything special here.

When something changes in the .[ch] files, how do you ensure it's not breaking something ?

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: Standalone libcrush: call for participation

[Ilya Dryomov]

On Mon, Jan 23, 2017 at 9:08 PM, Loic Dachary <loic@dachary.org> wrote:
> Hi Ilya,
>
> On 01/23/2017 07:43 PM, Ilya Dryomov wrote:
>> On Mon, Jan 23, 2017 at 6:00 PM, Loic Dachary <loic@dachary.org> wrote:
>>> Hi Adam,
>>>
>>> On 01/23/2017 05:29 PM, Adam C. Emerson wrote:
>>>> On 21/01/2017, Loic Dachary wrote:
>>>> [snip]
>>>>> The beginning of the next ceph release cycle seems like a good time
>>>>> to initiate that effort. If someone is willing to help with the
>>>>> bootstrap, now is a good time. The library will be hosted under the
>>>>> libcrush.org domain name. But that's about the only thing that
>>>>> happened so far :-)
>>>>
>>>> Would the Ceph git repository continue to have its own version of
>>>> CRUSH included, or would we pull in libcrush as a submodule?
>>>
>>> If all goes well and libcrush becomes actively and reliably maintained outside of the Ceph tree, it could either be a submodule or even a packaged dependency. Before that happens things can stay the way they are. I volunteer to manually monitor and cherry-pick whatever changes in the Ceph tree onto the standalone repository ( http://libcrush.org/main/libcrush/ ) so that it does not drift away.
>>>
>>>> Also, does this affect the relationship between userspace CRUSH and
>>>> Linux kernel CRUSH?
>>>
>>> I don't know :-) Unless I'm mistaken the code in the linux kernel is an unmodified copy of the code from src/crush/*.[ch].
>>
>> Correct.  libcrush won't impact the kernel in any way, as long as the
>> C core <-> C++ wrappers boundary stays intact, which it should ;)
>
> Yeah :-) Just rephrasing so I'm sure I understand you correctly: the important thing is that the C API stays as it is (functions & structs & constants) and that the semantic of each function is backward compatible. As long as it holds true, the kernel integration is going to be ok.

Well, that's a bit too strict.  Ideally, we need to able to cut
and-paste these files into the kernel with no or little effort.  The
API can change -- it just needs to be possible to bring the changes
over to the kernel client.

>
>> Some documentation will probably need to be added to cover things like
>> basic kernel compatibility (no floating point, stubs for asserts, etc)
>> and coding style for the C bits.
>
> Ack.
>
>> The core algorithm is far from where
>> libcrush initial activities would be centered though, I imagine.
>
> Indeed, more documentation & packaging right now.
>
>>> I however have no clue how it is tested to work as it should in both contexts. Other than running integration tests with teuthology.
>>>
>>> How do you suggest we go about it ?
>>
>> I don't think we need anything special here.
>
> When something changes in the .[ch] files, how do you ensure it's not breaking something ?

Run the k* test suites, mostly.  For invasive changes (like a new
tunable), instrument the kernel to dump the crushtool --show-mappings
output to the console and compare it with the .t, but that's a manual
process that requires writing some code.  Such changes happen around
twice a year, so it hasn't been a problem.

Thanks,

                Ilya

Re: Standalone libcrush: call for participation

[Ilya Dryomov]

On Mon, Jan 23, 2017 at 9:16 PM, Loic Dachary <loic@dachary.org> wrote:
>
>> Some documentation will probably need to be added to cover things like
>> basic kernel compatibility (no floating point, stubs for asserts, etc)
>> and coding style for the C bits.
>
> I added this
>
> http://libcrush.org/main/libcrush/blob/wip-2-doxygen/doc/contributing.md#linux-kernel
>
> Do you have anything else in mind ?

No need to try to list all the kernel quirks -- just point to
crush_compat.h.

These are the files that are shared:

- crush.[ch]
- crush_ln_table.h
- hash.[ch]
- mapper.[ch]

Everything else is fair game, notably builder.[ch].

Thanks,

                Ilya

Re: Standalone libcrush: call for participation

[Loic Dachary]

On 01/23/2017 10:44 PM, Ilya Dryomov wrote:
> On Mon, Jan 23, 2017 at 9:16 PM, Loic Dachary <loic@dachary.org> wrote:
>>
>>> Some documentation will probably need to be added to cover things like
>>> basic kernel compatibility (no floating point, stubs for asserts, etc)
>>> and coding style for the C bits.
>>
>> I added this
>>
>> http://libcrush.org/main/libcrush/blob/wip-2-doxygen/doc/contributing.md#linux-kernel
>>
>> Do you have anything else in mind ?
> 
> No need to try to list all the kernel quirks -- just point to
> crush_compat.h.
> 
> These are the files that are shared:
> 
> - crush.[ch]
> - crush_ln_table.h
> - hash.[ch]
> - mapper.[ch]
> 
> Everything else is fair game, notably builder.[ch].

Understood :-)

Is this http://libcrush.org/main/libcrush/blob/wip-2-doxygen/doc/mainpage.dox#L47 better ?

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: Standalone libcrush: call for participation

[Ilya Dryomov]

On Mon, Jan 23, 2017 at 10:59 PM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 01/23/2017 10:44 PM, Ilya Dryomov wrote:
>> On Mon, Jan 23, 2017 at 9:16 PM, Loic Dachary <loic@dachary.org> wrote:
>>>
>>>> Some documentation will probably need to be added to cover things like
>>>> basic kernel compatibility (no floating point, stubs for asserts, etc)
>>>> and coding style for the C bits.
>>>
>>> I added this
>>>
>>> http://libcrush.org/main/libcrush/blob/wip-2-doxygen/doc/contributing.md#linux-kernel
>>>
>>> Do you have anything else in mind ?
>>
>> No need to try to list all the kernel quirks -- just point to
>> crush_compat.h.
>>
>> These are the files that are shared:
>>
>> - crush.[ch]
>> - crush_ln_table.h
>> - hash.[ch]
>> - mapper.[ch]
>>
>> Everything else is fair game, notably builder.[ch].
>
> Understood :-)
>
> Is this http://libcrush.org/main/libcrush/blob/wip-2-doxygen/doc/mainpage.dox#L47 better ?

Looks good to me.

Thanks,

                Ilya