Backports Archive on lore.kernel.org
 help / color / Atom feed
From: Luis Chamberlain <mcgrof@kernel.org>
To: "Enrico Weigelt, metux IT consult" <lkml@metux.net>
Cc: Christoph Hellwig <hch@lst.de>,
	Brendan Higgins <brendanhiggins@google.com>,
	Cristina Moraru <cristina.moraru09@gmail.com>,
	"vegard.nossum@gmail.com" <vegard.nossum@gmail.com>,
	Valentin Rothberg <valentinrothberg@gmail.com>,
	Hannes Reinecke <hare@suse.de>, Sam Ravnborg <sam@ravnborg.org>,
	Michal Marek <mmarek@suse.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Tom Gundersen <teg@jklm.no>, Kay Sievers <kay@vrfy.org>,
	Rusty Russell <rusty@rustcorp.com.au>,
	Andrew Morton <akpm@linux-foundation.org>,
	backports@vger.kernel.org, Guenter Roeck <linux@roeck-us.net>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"rafael.j.wysocki" <rafael.j.wysocki@intel.com>,
	Dmitry Torokhov <dmitry.torokhov@gmail.com>,
	Takashi Iwai <tiwai@suse.de>,
	Mauro Carvalho Chehab <mchehab@osg.samsung.com>,
	Johannes Berg <johannes@sipsolutions.net>,
	Hauke Mehrtens <hauke@hauke-m.de>,
	Paul Bolle <pebolle@tiscali.nl>,
	Paul Gortmaker <paul.gortmaker@windriver.com>,
	Alexey Khoroshilov <khoroshilov@ispras.ru>,
	Sathya Prakash Veerichetty <sathya.prakash@broadcom.com>,
	"Martin K. Petersen" <martin.petersen@oracle.com>,
	Laurence Oberman <loberman@redhat.com>,
	Johannes Thumshirn <jthumshirn@suse.de>,
	Tejun Heo <tj@kernel.org>,
	Jej B <James.Bottomley@hansenpartnership.com>,
	Theodore Ts'o <tytso@mit.edu>,
	Daniel Jonsson <danijons@student.chalmers.se>,
	Andrzej Wasowski <wasowski@itu.dk>
Subject: Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
Date: Wed, 3 Jul 2019 22:42:34 +0000
Message-ID: <20190703224234.GY19023@42.do-not-panic.com> (raw)
In-Reply-To: <9a2ae341-9ea7-d4c6-7c3e-b12bb6515905@metux.net>

On Wed, Jul 03, 2019 at 09:31:33PM +0200, Enrico Weigelt, metux IT consult wrote:
> On 03.07.19 19:35, Luis Chamberlain wrote:
> 
> Hi,
> 
> >> Okay, but IIRC this will add more boilerplate those modules.
> > 
> > Just one module attribute.
> 
> Yes, but still one per module. This raises the question whether
> maintainers are willing to cope w/ tons of tiny patches for just
> one line - for something that will take quite some time to become
> actually useful (doesn't help much if only few drivers support it),
> and is only helpful in a few use cases.

This can be wrapped in kconfig and disabled by default.

> And to make it really useful, we also need some way to automatically
> derive which other symbols to enable (subsystems, etc), w/o auto-
> enabling stuff one doesn't need here. (are the defaults sane for
> this usecase ?)

No, that's a separate problem.
> 
> The main problem here, IMHO, is that the kconfig system doesn't really
> know what makes up a module (it only knows that something w =y cant
> depend on something thats =m).
> 
> So it smalls like we'd need some config language that really understands
> things like modules, subsystems, arches, etc with their properties and
> is used by both kconfig and kbuild. Then we could put all metadata there
> instead of the current macro calls. At that point we could also put
> things like match tables in here, which would solve the problem of
> finding the right driver by hardware descriptions.
> 
> But that's really a *big* topic, it's not easy.

Christoph's sugggestion goes along the lines with addressing some of
this, yet 2 years havegone by and no one is working on it and its hugely
intrusive. I'd welcome the patches.

> >> And I wonder whether target binaries are the right place for those
> >> things at all - IMHO that's something one wants to derive from the
> >> source code  / .config's.
> > 
> > For the use cases mentioned for why the module attribute is being
> > suggested it would help to not have to download kernel sources. 
> 
> Are we still talking about compiling custom kernels ?
> (how to do that w/o source code ?)

There is a difference between getting kernel sources for your current
running kernel Vs getting kernel sources from say Linus' tree or the
stable tree, or whatever subsystem you work on. I'm saying that with
a module attribute being present you would *not* have to get the kernel
sources for the current kernel you are running.

> > The only question we want to answer is: for the hardware components
> > present on this system, which configs options do I need to enable
> > to support these components?
> 
> What else would one need that data, if not compiling a custom kernel
> (which in turn needs the source) ?

One is compiling a custom kernel.

> > At least for virtualization we decided to support at least these two to
> > help:
> > 
> >   * make kvmconfig
> >   * make xenconfig
> 
> These two are rather simple. Most times there isn't much variance in
> virtual hardware (unless one starts directly mapping in pci or usb
> devices ...)

Actually no, there is huge variance on the qemu level and that problem
is not solved by the above.

> > Similar problem would be found if one wanted to find a desirable kernel
> > config for a remote system. One should be able to somehow scrape some
> > hardware information, dump that to a file, and then somehow generate
> > a working config for that system.
> 
> Yes. That's actually pretty much the same usecase (in my case I'd also
> have dts, lspci/lsusb output, etc)
> 
> > The module attribute being suggested would enable at least one way
> > to gather some of the required config symbols: symbols for *hardware*
> > and where one can run a modern kernel, with many features / hardware
> > enabled already.
> 
> But only for a pretty specific usecase.

Two: build time, and backports.

> I'm not opposed to this, but
> I wonder whether maintainers are willing to accept that stuff for just
> that specific usecase.

This is why we are discussing this on this thread.

Since we have 'make localmodconfig' already upstream, and since this
would help both users of that and backports I'd argue it makes sense
upstream. Otherwise I find it that having upstream 'make localmodconfig'
but not wanting to improve that problem space rather odd.

> > However, folks producing embedded systems *do* / *should* have a lot of
> > knowledge of their systems, and so the type of scheme you have devised
> > seems sensible for it.
> 
> Usually we have (unless we need to do reverse engineering :o). But it's
> a pretty time-consuming task. Especially if the requirements change
> several times in the development or lifetime of a specific product.
> 
> For example "oh, we now need eth", "naah, we don't wanna use usb
> anymore", "let's take a different SoM", ... not that have pretty
> orthogonal sets of configs we need to maintain: hardware- and non-
> hardware-related ones. And hardware-related ones can fall into different
> categories like fixed-attached/onboard vs. hotpluggable ones.
> 
> Recently I had a case where the customer requested xattr support, so
> I had to enable general xattr support as well as per-filesystem.
> Pretty simple, but having lots of those cases quickly sums up. One of
> the reasons why I've written my own little config generator.

I agree that the problem space you are dealing with should be made
easier.

> >> In embedded world, we often have scenarios where we want a really
> >> minimal kernel, but need to enable/disable certain hi-level peripherals
> >> in the middle of the project (eg. "oh, we also need ethernet, but we
> >> wanna drop usb"). There we'll have to find out what actual chip is,
> >> its corresponding driver, required subsystems, etc, and also kick off
> >> everything we don't need anymore.
> > 
> > Right. One *should* be able to tell some tool, hey, here is the list of
> > my desirable .config options. Go and figure out what I need to make that
> > work and give me a resulting .config. Its not easy.
> 
> I think I've already got into a pretty usable state - at least for my
> projects. For now only supports a few boards and limited set of
> features, but patches are always welcomed :)

A solution upstream would be better ;)

> >> I've thought about implementing some actual dependency tracking
> >> (at least recording the auto-enabled symbols), but didn't expect that
> >> to become practically usable anytime soon,
> > 
> > The ability to easily ask the kernel to enable the components needed
> > for a respective config option *is* very useful but indeed not easy.
> 
> Yes, it would need to understand things like conditional definitions
> to deduce that certain things need to be enabled first, before certain
> drivers become choosable.
> 
> > This is not the only space where this problem exists. Similar problem
> > exists for distribution packages, and dependencies. Challenges have
> > been made for proper research towards these problems, and such research
> > has lead distributions to opt to enable some of these algorithms.
> 
> The problem w/ dependencies is that there can be different types of
> dependencies, as well as different types of software objects. Just
> solving the expressions is only a part of the problem.
> 
> > This begs the question if we could learn from similar efforts on Linux
> > for these sorts of questions. One possibility here is to evaluate the
> > prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> > (MUSes) support, which should be be able to address thir problem. This
> > prospect is ongoing and currrent R&D is active, for details refer to:
> > 
> > https://kernelnewbies.org/KernelProjects/kconfig-sat
> 
> Good tip, I'll have a look at it.
> 
> > It certainly can be useful for components, ie, not hardware. But for
> > hardware a one-to-one mapping of one driver to one config would be of
> > much more use.
> 
> Unfortunately, we don't have this 1:1 mapping. Often drivers support
> different sets of devices, depending on other factors, sometimes sub-
> options (eg. different hw versions), sometimes depending on other
> subsystems, sometimes arch-specific, etc, etc.
> 
> I think we should work towards that, but I doubt we'd reach that goal
> anytime soon, and begs the question whether it's really worth all the
> effort required for that.

Code speaks, and fortunately its being worked on.

> > It would be wonderful if for instance kconfig
> > supported a way to group a major set of components under *one* config
> > symbol and then say: "I want this major component enabled" and then it'd
> > go and enable all the defaults which would be required for it. 
> 
> Yes, thought about that, too. For example have syms for selecting whole
> boards and features of them - a bit like this:
> 
>   --> Preconfigure for specific boards
>       --> board A
>       --> board B
>       ...
>   --> Enable board features
>       --> Ethernet port
>       --> Display
>           --> Touch panel
>       --> Audio
>       ....
> 
> BUT: this would turn into maintenance hell, so I dropped that idea.
> 
> > An example is if you
> > wanted to enable PCI on a system which didn't support it. Because of
> > this, it seems you'd want *all* desirable configs and let a piece of
> > software figure out what you need / can enable. And.. this is precisely
> > where the SAT solver with MUSes could help...
> 
> Yes, but this piece of software first needs to know whether eg. PCI
> is available on that HW. Oh, and things like PCI could be a dependency
> as well as an feature on its own, depending on how you gonna use it.
> (eg. if directly access from userland or VMs).

Yes, but that is a separate problem. The way 'make localmodconfig'
addresses this is by keeping as modules things which are already
modules. And keeping things it found as =y as buit-in in your kernel as
well.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

  reply index

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 17:55 ` Luis R. Rodriguez
2016-08-19  9:07   ` Michal Marek
2016-08-22 19:48     ` Cristina-Gabriela Moraru
2016-08-23 21:32     ` Luis R. Rodriguez
2016-08-24 11:05       ` Michal Marek
2016-08-24 16:33         ` Luis R. Rodriguez
2016-08-24 17:31           ` Naveen Kumar
2016-08-22 19:35   ` Cristina-Gabriela Moraru
2016-08-23 19:17     ` Luis R. Rodriguez
2016-08-25  7:43   ` Christoph Hellwig
2016-08-25  8:00     ` Johannes Berg
2016-08-25 19:51       ` Luis R. Rodriguez
2016-08-25  8:41     ` Michal Marek
2016-08-25 20:19     ` Luis R. Rodriguez
2019-02-05 22:07       ` Luis Chamberlain
2019-06-26 22:21         ` Luis Chamberlain
2019-06-27  4:50           ` Christoph Hellwig
2019-06-28 18:40             ` Luis Chamberlain
2019-06-29  8:42               ` Greg Kroah-Hartman
2019-07-02 20:51                 ` Luis Chamberlain
2019-07-03  7:40                   ` Greg Kroah-Hartman
2019-07-03 16:50                     ` Luis Chamberlain
2019-07-03 18:57                       ` Greg Kroah-Hartman
2019-07-03 22:25                         ` Luis Chamberlain
2019-07-11 23:07                           ` Brendan Higgins
2019-07-11 23:22                             ` Luis Chamberlain
2019-07-03 12:16               ` Enrico Weigelt, metux IT consult
2019-07-03 17:35                 ` Luis Chamberlain
2019-07-03 19:31                   ` Enrico Weigelt, metux IT consult
2019-07-03 22:42                     ` Luis Chamberlain [this message]
2019-07-11 23:27                       ` Brendan Higgins
2019-07-13 14:44                         ` Enrico Weigelt, metux IT consult
     [not found] ` <1471462023-119645-3-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:10   ` [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter Luis R. Rodriguez
2016-08-18 18:55     ` Luis R. Rodriguez
2016-08-20 15:11     ` Cristina-Gabriela Moraru
2016-08-23 19:07       ` Luis R. Rodriguez
     [not found] ` <1471462023-119645-2-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:22   ` [RFC PATCH 1/5] Add generation of Module.symb in streamline_config Luis R. Rodriguez
2016-08-18 18:32     ` Luis R. Rodriguez
     [not found]     ` <CAGZ2q2xi9Uy-ye387=mWhy_fOEJBC593Nos7fH027m-_ZdoOXA@mail.gmail.com>
2016-08-20 14:49       ` Cristina-Gabriela Moraru
2016-08-23 19:00       ` Luis R. Rodriguez
     [not found] ` <1471462023-119645-4-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:30   ` [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile Luis R. Rodriguez
     [not found] ` <1471462023-119645-5-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:59   ` [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute Luis R. Rodriguez
2016-08-20 15:16     ` Cristina-Gabriela Moraru
2016-08-23 19:10       ` Luis R. Rodriguez
     [not found] ` <1471462023-119645-6-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 19:02   ` [RFC PATCH 5/5] Add kconf_symb as kernel " Luis R. Rodriguez

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190703224234.GY19023@42.do-not-panic.com \
    --to=mcgrof@kernel.org \
    --cc=James.Bottomley@hansenpartnership.com \
    --cc=akpm@linux-foundation.org \
    --cc=backports@vger.kernel.org \
    --cc=brendanhiggins@google.com \
    --cc=cristina.moraru09@gmail.com \
    --cc=danijons@student.chalmers.se \
    --cc=dmitry.torokhov@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=hare@suse.de \
    --cc=hauke@hauke-m.de \
    --cc=hch@lst.de \
    --cc=johannes@sipsolutions.net \
    --cc=jthumshirn@suse.de \
    --cc=kay@vrfy.org \
    --cc=khoroshilov@ispras.ru \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux@roeck-us.net \
    --cc=lkml@metux.net \
    --cc=loberman@redhat.com \
    --cc=martin.petersen@oracle.com \
    --cc=mchehab@osg.samsung.com \
    --cc=mmarek@suse.com \
    --cc=paul.gortmaker@windriver.com \
    --cc=pebolle@tiscali.nl \
    --cc=rafael.j.wysocki@intel.com \
    --cc=rusty@rustcorp.com.au \
    --cc=sam@ravnborg.org \
    --cc=sathya.prakash@broadcom.com \
    --cc=teg@jklm.no \
    --cc=tiwai@suse.de \
    --cc=tj@kernel.org \
    --cc=tytso@mit.edu \
    --cc=valentinrothberg@gmail.com \
    --cc=vegard.nossum@gmail.com \
    --cc=wasowski@itu.dk \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Backports Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/backports/0 backports/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 backports backports/ https://lore.kernel.org/backports \
		backports@vger.kernel.org backports@archiver.kernel.org
	public-inbox-index backports


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.backports


AGPL code for this site: git clone https://public-inbox.org/ public-inbox