kconfig as a git subtree on Linux

kconfig as a git subtree on Linux

[Luis Chamberlain]

I'd like to propose we discuss the possibility of taking kconfig and
making it a git subtree under the Linux kernel. This would allow
other projects outside of the Linux kernel to be able to update their
own copy / fork of kconfig in a jiffie *very* easily.

Why? kconfig has far outlived its own purpose as a modeling variablity
language hack on the Linux kernel, and *is* forked and *used* by *many*
projects, to the point I think ignoring its use outside of the Linux
kernel is doing a disservice to its own growth. Counting just personal
projects I have 3 projects which use kconfig alone. Last time I counted
about 30 or so external projects. I'm confident this is a shy number
of today's reality.

Yes this does beg the question, that if this is done, can / should
this be considered elsewhere. And yes, I'd like to hear the rants
about why this would be a completely unacceptable practice *at all*.

I've been using git subtrees now for another project and I'm *very*
happy with it so far. It lets me keep a project with some code on
its own git tree, and then multiple third party trees can embrace
it, and decide to update later whenever they want. An example is the
update_ssh_config [0] python script which I use on kdevops [1] for
vagrant, but since it is also used for terraform and terraform uses
modules I ended up sharing the code for terraform for its own
terrarorm module [2] [3]. I do most development and unit testing
on the main update_ssh_config git tree, and when I want to integrate
its changes into kdevops I just run:

make refresh

This is because on kdevops I have:

$ cat Makefile.subtrees
# If you need to use a git subtree, please add it here.
add-remotes:
	git remote add update_ssh_config https://github.com/mcgrof/update_ssh_config.git

add-commits:
	git subtree add --prefix=playbooks/roles/update_ssh_config_vagrant/update_ssh_config update_ssh_config master

refresh:
	git fetch update_ssh_config git subtree pull --prefix=playbooks/roles/update_ssh_config_vagrant/update_ssh_config update_ssh_config master

Likewise for my terraform module, however there I just have these
targets on my make Makefile. A person who first git cloned either the
kdevops tree of the terraform module tree would first have to run the
targets:

   * make add-remotes
   * make add-commits

Today's process for updating kconfig on external projects requires
substrantial manual oversight.

[0] https://github.com/mcgrof/update_ssh_config
[1] https://github.com/mcgrof/kdevops/
[2] https://github.com/mcgrof/terraform-kdevops-add-host-ssh-config
[3] https://registry.terraform.io/modules/mcgrof/add-host-ssh-config/kdevops/latest

  Luis

Re: kconfig as a git subtree on Linux

[Christoph Hellwig]

On Wed, Nov 25, 2020 at 05:25:44PM +0000, Luis Chamberlain wrote:
> I'd like to propose we discuss the possibility of taking kconfig and
> making it a git subtree under the Linux kernel. This would allow
> other projects outside of the Linux kernel to be able to update their
> own copy / fork of kconfig in a jiffie *very* easily.

Hell no.  Not because I'm against sharing, but because git submodules
are a complete fucking pain to deal with.  Adding submodules just fucks
up being able to easily use a repository.  

Re: kconfig as a git subtree on Linux

[Luis Chamberlain]

On Thu, Nov 26, 2020 at 09:13:29AM +0000, Christoph Hellwig wrote:
> On Wed, Nov 25, 2020 at 05:25:44PM +0000, Luis Chamberlain wrote:
> > I'd like to propose we discuss the possibility of taking kconfig and
> > making it a git subtree under the Linux kernel. This would allow
> > other projects outside of the Linux kernel to be able to update their
> > own copy / fork of kconfig in a jiffie *very* easily.
> 
> Hell no.  Not because I'm against sharing, but because git submodules
> are a complete fucking pain to deal with.  Adding submodules just fucks
> up being able to easily use a repository.  

I agree wholeheartedly! But I was not alking about git submodules. I
think git submodules should be taken to the back of an alley and shot.
But, git subtrees, however, now that is proving to me to be very sexy.

  Luis

Re: kconfig as a git subtree on Linux

[Boris Kolpackov]

Luis Chamberlain <mcgrof@kernel.org> writes:

> I'd like to propose we discuss the possibility of taking kconfig and
> making it a git subtree under the Linux kernel. This would allow
> other projects outside of the Linux kernel to be able to update their
> own copy / fork of kconfig in a jiffie *very* easily.

I am maintaining one such copy/fork[1] and for me the effort to pull
in the new version of upstream (which I currently do by just copying
scripts/kconfig/*) is nothing compared to the effort of maintaining
a set of patches[2] on top of that which are necessary to make kconfig
buildable on other platforms and usable with other build systems.

So unless there is also an agreement that such portability patches
are now welcome, this is not going to be a major improvement for me.
And right now such patches are clearly not welcome[3] (but no hard
feelings; I wouldn't touch Windows with a ten-foot pole if I could
help it).

[1] https://github.com/build2-packaging/kconfig
[2] https://github.com/build2-packaging/kconfig/commits/upstream-5.10-rc1
[3] https://www.spinics.net/lists/linux-kbuild/msg27460.html

Re: kconfig as a git subtree on Linux

[Luis Chamberlain]

On Thu, Nov 26, 2020 at 12:38:41PM +0200, Boris Kolpackov wrote:
> Luis Chamberlain <mcgrof@kernel.org> writes:
> 
> > I'd like to propose we discuss the possibility of taking kconfig and
> > making it a git subtree under the Linux kernel. This would allow
> > other projects outside of the Linux kernel to be able to update their
> > own copy / fork of kconfig in a jiffie *very* easily.
> 
> I am maintaining one such copy/fork[1] and for me the effort to pull
> in the new version of upstream (which I currently do by just copying
> scripts/kconfig/*) is nothing compared to the effort of maintaining
> a set of patches[2] on top of that which are necessary to make kconfig
> buildable on other platforms and usable with other build systems.
> 
> So unless there is also an agreement that such portability patches
> are now welcome, this is not going to be a major improvement for me.

Unless you have tried git subtrees, I doubt you really mean this. How
is a 'make refresh' command as comparable as manually pulling in
changes from a project to your project?

> And right now such patches are clearly not welcome[3] (but no hard
> feelings; I wouldn't touch Windows with a ten-foot pole if I could
> help it).

Portability of kconfig to other platorm is a topic of its own. If that
sort of conversation can exist, I think it would have to be *secondary*
to deciding whether or not kconfig lives on its own to allow other
Linux projects to benefit from it.

  Luis

Re: kconfig as a git subtree on Linux

[Masahiro Yamada]

On Thu, Nov 26, 2020 at 2:25 AM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> I'd like to propose we discuss the possibility of taking kconfig and
> making it a git subtree under the Linux kernel. This would allow
> other projects outside of the Linux kernel to be able to update their
> own copy / fork of kconfig in a jiffie *very* easily.
>
> Why? kconfig has far outlived its own purpose as a modeling variablity
> language hack on the Linux kernel, and *is* forked and *used* by *many*
> projects, to the point I think ignoring its use outside of the Linux
> kernel is doing a disservice to its own growth. Counting just personal
> projects I have 3 projects which use kconfig alone. Last time I counted
> about 30 or so external projects. I'm confident this is a shy number
> of today's reality.
>
> Yes this does beg the question, that if this is done, can / should
> this be considered elsewhere. And yes, I'd like to hear the rants
> about why this would be a completely unacceptable practice *at all*.
>
> I've been using git subtrees now for another project and I'm *very*
> happy with it so far. It lets me keep a project with some code on
> its own git tree, and then multiple third party trees can embrace
> it, and decide to update later whenever they want. An example is the
> update_ssh_config [0] python script which I use on kdevops [1] for
> vagrant, but since it is also used for terraform and terraform uses
> modules I ended up sharing the code for terraform for its own
> terrarorm module [2] [3]. I do most development and unit testing
> on the main update_ssh_config git tree, and when I want to integrate
> its changes into kdevops I just run:
>
> make refresh
>
> This is because on kdevops I have:
>
> $ cat Makefile.subtrees
> # If you need to use a git subtree, please add it here.
> add-remotes:
>         git remote add update_ssh_config https://github.com/mcgrof/update_ssh_config.git
>
> add-commits:
>         git subtree add --prefix=playbooks/roles/update_ssh_config_vagrant/update_ssh_config update_ssh_config master
>
> refresh:
>         git fetch update_ssh_config git subtree pull --prefix=playbooks/roles/update_ssh_config_vagrant/update_ssh_config update_ssh_config master
>
> Likewise for my terraform module, however there I just have these
> targets on my make Makefile. A person who first git cloned either the
> kdevops tree of the terraform module tree would first have to run the
> targets:
>
>    * make add-remotes
>    * make add-commits
>
> Today's process for updating kconfig on external projects requires
> substrantial manual oversight.
>
> [0] https://github.com/mcgrof/update_ssh_config
> [1] https://github.com/mcgrof/kdevops/
> [2] https://github.com/mcgrof/terraform-kdevops-add-host-ssh-config
> [3] https://registry.terraform.io/modules/mcgrof/add-host-ssh-config/kdevops/latest
>
>   Luis




I want to keep Kconfig in the kernel tree.

Other projects can throw in scripts/kconfig/,
but in some cases, they might need some adjustments
for their use.



Kconfig is almost standalone, but in reality,
there are some interactions between the build system
and kconfig.


I want to reserve the freedom for optimizations
that span across both kbuild and kconfig.

For example,

058507195b534e6a1b3e8bf716b816ecf090c9c3
8b41fc4454e36fbfdbb23f940d023d4dece2de29

If we split Kconfig out as a standalone project,
such optimizations would become difficult.




A long time has passed since the Kconfig introduction, but
the Kconfig implementation is still in an immature state, in my opinion.


I want to fix them, and some of them are done
in a way that breaks backward compatibility.


In such cases, I fix in-kernel Kconfig files first,
and then, I fix the kconfig implementation.

For example,

Fix users:   8636a1f9677db4f883f29a072f401303acfc2edd
Fix Kconfig: f5451582c4e22ce8912aae4950810f3598c9b516


Fix users:   df8df5e4bc37e39010cfdf5d50cf726fe08aae5b
Fix Kconfig: 09d5873e4d1f70202314b5fe40160f9b14b9d2d0


I can proceed confidently because we have Kconfig files
and scripts/kconfig in the same tree, and I can
do build tests easily.


Again, if we split Kconfig as a standalone project,
I would need to care about which projects would be
affected by such changes, say, not only Linux but also
dozens of projects, which I do not know much about.






-- 
Best Regards
Masahiro Yamada