mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Ernesto Puerta]

[sent both to ceph-devel and ceph-users lists, as it might be of
interest for both audiences]

Hi all,

This e-mail is just to announce the WIP on backporting dashboard_v2
(http://docs.ceph.com/docs/master/mgr/dashboard/) from master to
Luminous release.

The ultimate goal for this backport is to replace dashboard_v1 in
Luminous and provide, **as much as possible**, (see note below) a
level of functionality on a par with master's (i. e.: RBD and RGW
management, HTTPS support, User Management, Role Based Access Control,
Grafana integration, SSO, etc.).

NOTE: This will be mostly driven as a best-effort (currently one-army
man) task, as long as Luminous is N-2 and backports to Mimic (N-1)
have officially ceased. Besides, Luminous lacks some features that
Ceph Dashboard v2 exposes. Therefore, the policy to follow for this
backport will be:

1.- Backport by cherry-picking individual commits, except for merge
(PR) commits that comprise a large amount of commits (e.g.: PR #23271
with 163 commits).

2.A- Do not bring any new functionality from master to Luminous other
than dashboard_v2 (commit subject: {mgr,doc,test}/dashboard{,_v2},
paths: {src/pybind/mgr/dashboard, qa/suites/rados/mgr/tasks/dashboard}
+ building/testing {CMakeLists.txt, ceph.spec.in, vstart.sh,
debian/rules, etc.})

2.B- When dashboard_v2 exposes or relies on some master-only
functionality, modify dashboard code in the most minimal way to handle
that (e.g.: disable RBD OPERATIONS FEATURE,
https://github.com/ceph/ceph/commit/8479a91c4fa45b39ef5cf24a857751de3a5c630f)

3. Add the necessary 'glue' commits to ensure cherry-picked pieces of
code work smoothly (e.g.: due to different toolchain versions or
supported idioms, like the case of C++17, or other non-functional
deps).

The first chunk of this backporting is already in PR ready status
(https://ceph.com/planet/the-ceph-dashboard-v2-pull-request-is-ready-for-review/,
https://github.com/ceph/ceph/pull/23271,
http://tracker.ceph.com/issues/24788).

If anyone else wants to contribute either on the dev side, or as test
user (I may point you to the corresponding shaman builds), don't
hesitate to contact me (Github/IRC user: epuertat). For tracking the
progress of this activity or any further questions/suggestions, etc.,
you are welcome to show up at dashboard daily stand-ups or biweekly
sync meetings.

Kind Regards,

Ernesto

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Willem Jan Withagen]

On 22/08/2018 12:16, Ernesto Puerta wrote:
> [sent both to ceph-devel and ceph-users lists, as it might be of
> interest for both audiences]
> 
> Hi all,
> 
> This e-mail is just to announce the WIP on backporting dashboard_v2
> (http://docs.ceph.com/docs/master/mgr/dashboard/) from master to
> Luminous release.
> 
> The ultimate goal for this backport is to replace dashboard_v1 in
> Luminous and provide, **as much as possible**, (see note below) a
> level of functionality on a par with master's (i. e.: RBD and RGW
> management, HTTPS support, User Management, Role Based Access Control,
> Grafana integration, SSO, etc.).

If done so, I would prefer to also keep the old "simple" Dashboard.
Reason for that is the ease of portability.

The old dashboard does not need any package fetch while 
building/installing. Something that is not very handy when building 
FreeBSD packages. And I haven't gotten around to determining how to get 
around that.

Next to that: that dashboard is "simple". Something I really like, but 
that is perhaps personal.

Suggest renaming it to simpledash or dashboard_v1 and keep it in the tree.

Thanx,
--WjW

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Ernesto Puerta]

Thanks for your feedback, Willem!

> The old dashboard does not need any package fetch while
> building/installing. Something that is not very handy when building
> FreeBSD packages. And I haven't gotten around to determining how to > get around that.

I thought that https://github.com/ceph/ceph/pull/22562 fixed make-dist
issues on FreeBSD. Is that not working yet? Let us know if that's the
case!

> Suggest renaming it to simpledash or dashboard_v1 and keep it in the
> tree.

Unfortunately, keeping v1 is not as simple as moving the dashboard to
a separate directory (unless we leave it hanging as dead code).
Dashboard_v2 completely replaces dashboard_v1, and that also means
unit test, QA suites, and references in common files (install-deps.sh,
CMakeLists.txt, ceph.spec.in, debian/*, do_freebsd.sh, vstart.sh,
etc.).

My concern is that properly keeping both ones would go beyond a
long-but-mostly-clean cherry-picking. It'd involve Luminous actively
diverging from master, which might burden other backports with
manual/creative conflict-solving.

KR,

Ernesto

On Wed, Aug 22, 2018 at 12:43 PM Willem Jan Withagen <wjw-dOtk1Lsa4IaEVqv0pETR8A@public.gmane.org> wrote:
>
> On 22/08/2018 12:16, Ernesto Puerta wrote:
> > [sent both to ceph-devel and ceph-users lists, as it might be of
> > interest for both audiences]
> >
> > Hi all,
> >
> > This e-mail is just to announce the WIP on backporting dashboard_v2
> > (http://docs.ceph.com/docs/master/mgr/dashboard/) from master to
> > Luminous release.
> >
> > The ultimate goal for this backport is to replace dashboard_v1 in
> > Luminous and provide, **as much as possible**, (see note below) a
> > level of functionality on a par with master's (i. e.: RBD and RGW
> > management, HTTPS support, User Management, Role Based Access Control,
> > Grafana integration, SSO, etc.).
>
> If done so, I would prefer to also keep the old "simple" Dashboard.
> Reason for that is the ease of portability.
>
> The old dashboard does not need any package fetch while
> building/installing. Something that is not very handy when building
> FreeBSD packages. And I haven't gotten around to determining how to get
> around that.
>
> Next to that: that dashboard is "simple". Something I really like, but
> that is perhaps personal.
>
> Suggest renaming it to simpledash or dashboard_v1 and keep it in the tree.
>
> Thanx,
> --WjW
>

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[David Turner]

[-- Attachment #1.1: Type: text/plain, Size: 3309 bytes --]

My initial reaction to this PR/backport was questioning why such a major
update would happen on a dot release of Luminous.  Your reaction to keeping
both dashboards viable goes to support that.  Should we really be
backporting features into a dot release that force people to change how
they use the software?  That seems more of the purpose of having new
releases.

I haven't really used either dashboard though.  Other than adding admin
functionality, does it remove any functionality of the previous dashboard?

On Wed, Aug 22, 2018 at 1:43 PM Ernesto Puerta <epuertat-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Thanks for your feedback, Willem!
>
> > The old dashboard does not need any package fetch while
> > building/installing. Something that is not very handy when building
> > FreeBSD packages. And I haven't gotten around to determining how to >
> get around that.
>
> I thought that https://github.com/ceph/ceph/pull/22562 fixed make-dist
> issues on FreeBSD. Is that not working yet? Let us know if that's the
> case!
>
> > Suggest renaming it to simpledash or dashboard_v1 and keep it in the
> > tree.
>
> Unfortunately, keeping v1 is not as simple as moving the dashboard to
> a separate directory (unless we leave it hanging as dead code).
> Dashboard_v2 completely replaces dashboard_v1, and that also means
> unit test, QA suites, and references in common files (install-deps.sh,
> CMakeLists.txt, ceph.spec.in, debian/*, do_freebsd.sh, vstart.sh,
> etc.).
>
> My concern is that properly keeping both ones would go beyond a
> long-but-mostly-clean cherry-picking. It'd involve Luminous actively
> diverging from master, which might burden other backports with
> manual/creative conflict-solving.
>
> KR,
>
> Ernesto
>
> On Wed, Aug 22, 2018 at 12:43 PM Willem Jan Withagen <wjw-dOtk1Lsa4IaEVqv0pETR8A@public.gmane.org>
> wrote:
> >
> > On 22/08/2018 12:16, Ernesto Puerta wrote:
> > > [sent both to ceph-devel and ceph-users lists, as it might be of
> > > interest for both audiences]
> > >
> > > Hi all,
> > >
> > > This e-mail is just to announce the WIP on backporting dashboard_v2
> > > (http://docs.ceph.com/docs/master/mgr/dashboard/) from master to
> > > Luminous release.
> > >
> > > The ultimate goal for this backport is to replace dashboard_v1 in
> > > Luminous and provide, **as much as possible**, (see note below) a
> > > level of functionality on a par with master's (i. e.: RBD and RGW
> > > management, HTTPS support, User Management, Role Based Access Control,
> > > Grafana integration, SSO, etc.).
> >
> > If done so, I would prefer to also keep the old "simple" Dashboard.
> > Reason for that is the ease of portability.
> >
> > The old dashboard does not need any package fetch while
> > building/installing. Something that is not very handy when building
> > FreeBSD packages. And I haven't gotten around to determining how to get
> > around that.
> >
> > Next to that: that dashboard is "simple". Something I really like, but
> > that is perhaps personal.
> >
> > Suggest renaming it to simpledash or dashboard_v1 and keep it in the
> tree.
> >
> > Thanx,
> > --WjW
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 4512 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Kai Wagner]

[-- Attachment #1.1.1: Type: text/plain, Size: 308 bytes --]

On 22.08.2018 20:57, David Turner wrote:
> does it remove any functionality of the previous dashboard?
No it doesn't. All dashboard_v1 features are integrate and part of the
dashboard_v2 as well.

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Lenz Grimmer]

[-- Attachment #1.1.1: Type: text/plain, Size: 1402 bytes --]

On 08/22/2018 08:57 PM, David Turner wrote:

> My initial reaction to this PR/backport was questioning why such a
> major update would happen on a dot release of Luminous.  Your
> reaction to keeping both dashboards viable goes to support that.
> Should we really be backporting features into a dot release that
> force people to change how they use the software?  That seems more of
> the purpose of having new releases.

This is indeed an unusual case. But considering that the Dashboard does
not really change any of the Ceph core functionality but adds a lot of
value by improving the usability and manageability of Ceph, we agreed
with Sage on making an exception here.

> I haven't really used either dashboard though.  Other than adding
> admin functionality, does it remove any functionality of the previous
> dashboard?

Like Kai wrote, our initial goal was to reach feature parity with
Dashboard v1, in order to not introduce a regression when replacing it.

In the meanwhile, Dashboard v2 is way beyond that and we have added a
lot of additional functionality, e.g. RBD and RGW management.

With backporting this to Luminous, we also hope to reach a larger
audience of users that have not updated to Mimic yet.

Lenz

-- 
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG Nürnberg)


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Willem Jan Withagen]

On 22/08/2018 19:42, Ernesto Puerta wrote:
> Thanks for your feedback, Willem!
> 
>> The old dashboard does not need any package fetch while
>> building/installing. Something that is not very handy when building
>> FreeBSD packages. And I haven't gotten around to determining how to > get around that.
> 
> I thought that https://github.com/ceph/ceph/pull/22562 fixed make-dist
> issues on FreeBSD. Is that not working yet? Let us know if that's the
> case!

Eh, yes and no....
The PR allows me to build dashboard in tree. All the guarding with 
nodeenv and such are not there. So that is already less clean than it 
should be, but it works for testing.

Now building a FreeBSD package is a totally different cookie.
That does not work with the way it is now designed.
FreeBSD package building does not allow data to be fetch half way thru 
the build process!!! So calling something like npm during building is taboo.

There is a fetch stage, where al the sources and submodules are fetched.
And I have 2 make targets: pre_fetch and post_fetch where I can get 
extra sources that are required.
Problem is also that the sources are only unpacked AFTER post_fetch, so
even running a part of the Cmake code in post_fetch does not work, 
because the tree is not there.

And it hard to track what npm is al installing, because it seems it is 
pulling in a complete forrest of new dependancies.

So my way out at the moment is probably going to be:
	offline build the tree,
	take all the dashboard resulting source.
		including what gets installed in /usr/local. :(
	put it in a blob
	put the blob in a FreeBSD port where I fetch the blob
	make a package from that.
and then we have net/ceph-dashboard. :(
I did not mention that I need to do this for 2-3 releases 8-|

>> Suggest renaming it to simpledash or dashboard_v1 and keep it in the
>> tree.
> 
> Unfortunately, keeping v1 is not as simple as moving the dashboard to
> a separate directory (unless we leave it hanging as dead code).
> Dashboard_v2 completely replaces dashboard_v1, and that also means
> unit test, QA suites, and references in common files (install-deps.sh,
> CMakeLists.txt, ceph.spec.in, debian/*, do_freebsd.sh, vstart.sh,
> etc.).
> 
> My concern is that properly keeping both ones would go beyond a
> long-but-mostly-clean cherry-picking. It'd involve Luminous actively
> diverging from master, which might burden other backports with
> manual/creative conflict-solving.

Yes, I understand the concerns. But mine are not less complicated.
And especially if all the parts of v1 are also in v2, it requires "just" 
a smart division of tests ao. So that v1 can be tested as a subset.
I know easier said then done.

--WjW

> KR,
> 
> Ernesto
> 
> On Wed, Aug 22, 2018 at 12:43 PM Willem Jan Withagen <wjw-dOtk1Lsa4IaEVqv0pETR8A@public.gmane.org> wrote:
>>
>> On 22/08/2018 12:16, Ernesto Puerta wrote:
>>> [sent both to ceph-devel and ceph-users lists, as it might be of
>>> interest for both audiences]
>>>
>>> Hi all,
>>>
>>> This e-mail is just to announce the WIP on backporting dashboard_v2
>>> (http://docs.ceph.com/docs/master/mgr/dashboard/) from master to
>>> Luminous release.
>>>
>>> The ultimate goal for this backport is to replace dashboard_v1 in
>>> Luminous and provide, **as much as possible**, (see note below) a
>>> level of functionality on a par with master's (i. e.: RBD and RGW
>>> management, HTTPS support, User Management, Role Based Access Control,
>>> Grafana integration, SSO, etc.).
>>
>> If done so, I would prefer to also keep the old "simple" Dashboard.
>> Reason for that is the ease of portability.
>>
>> The old dashboard does not need any package fetch while
>> building/installing. Something that is not very handy when building
>> FreeBSD packages. And I haven't gotten around to determining how to get
>> around that.
>>
>> Next to that: that dashboard is "simple". Something I really like, but
>> that is perhaps personal.
>>
>> Suggest renaming it to simpledash or dashboard_v1 and keep it in the tree.
>>
>> Thanx,
>> --WjW
>>

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Willem Jan Withagen]

On 23/08/2018 11:22, Lenz Grimmer wrote:
> On 08/22/2018 08:57 PM, David Turner wrote:
> 
>> My initial reaction to this PR/backport was questioning why such a
>> major update would happen on a dot release of Luminous.  Your
>> reaction to keeping both dashboards viable goes to support that.
>> Should we really be backporting features into a dot release that
>> force people to change how they use the software?  That seems more of
>> the purpose of having new releases.
> 
> This is indeed an unusual case. But considering that the Dashboard does
> not really change any of the Ceph core functionality but adds a lot of
> value by improving the usability and manageability of Ceph, we agreed
> with Sage on making an exception here.
> 
>> I haven't really used either dashboard though.  Other than adding
>> admin functionality, does it remove any functionality of the previous
>> dashboard?
> 
> Like Kai wrote, our initial goal was to reach feature parity with
> Dashboard v1, in order to not introduce a regression when replacing it.
> 
> In the meanwhile, Dashboard v2 is way beyond that and we have added a
> lot of additional functionality, e.g. RBD and RGW management.
> 
> With backporting this to Luminous, we also hope to reach a larger
> audience of users that have not updated to Mimic yet.

Wierd approach question....

Is there still some "logic" separation between the v1 and v2 stuff?
And is it possible to build what is used to be v1 without importing the 
npm stuff? That way I can "easily" package and distribute that piece.

ATM this plan really cripples my possibilities to package a bit of 
dashboard... where L and M did have something workable.

Just keeping it in the tree as "dead wood" for the time being would help 
me get people a taste of what is there. Just ripping it out does not 
help. And I'll try to work out a solution before the release of 
Nautilus, or accept that it will be gone and has to be v2.

--WjW

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Ernesto Puerta]

Thanks all for sharing your views, and thanks to Lenz & Kai for the
clarifications.

For those, like David, not familiar with dashboard_v2 (or even with
dashboard_v1), you may check this short clip
(https://youtu.be/m5i3x4eR6k4), which goes through the dashboard_v2 as
per this first backport (feature par with v1).

As Lenz said, the improvement in operability this brings in to
Luminous largely exceed its cost (doable yet tough) and risks (limited
as long as dashboard_v2 is a ceph-mgr python module), and makes this
move worthwhile for both new and existing Ceph users.

@Willem, given your comments come from a technical ground, let's
address those technically. As you say, dashboard_v2 is already in
Mimic and will be soon in Nautilus when released, so for FreeBSD the
issue will anyhow be there. Let's look for a technical solution (both
short and long-term): shall we have meeting to handle the FreeBSD
specifics?

Kind Regards,
Ernesto

Re: mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

[Willem Jan Withagen]

On 23/08/2018 12:47, Ernesto Puerta wrote:

> @Willem, given your comments come from a technical ground, let's
> address those technically. As you say, dashboard_v2 is already in
> Mimic and will be soon in Nautilus when released, so for FreeBSD the
> issue will anyhow be there. Let's look for a technical solution (both
> short and long-term): shall we have meeting to handle the FreeBSD
> specifics?

I have bit more time than that...
Usually I start makeing a port once the release is stable, so in this 
case 14.2.x. Becaue the FreeBSD users that like to live dangerously 
usually can also build their own source from the repo.

But you right, one day I need to bite the bullit.

ATM I'm contemplating all kinds of scenarios to make it workable for all 
participants: devs, porters, users. And I haven't really found the 
golden egg.
So yes, lets take it private and talk.

--WjW