Backports Archive on lore.kernel.org
 help / Atom feed
* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
       [not found] <1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com>
@ 2016-08-18 17:55 ` Luis R. Rodriguez
  2016-08-19  9:07   ` Michal Marek
                     ` (2 more replies)
       [not found] ` <1471462023-119645-3-git-send-email-cristina.moraru09@gmail.com>
                   ` (4 subsequent siblings)
  5 siblings, 3 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 17:55 UTC (permalink / raw)
  To: Cristina Moraru, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, Michal Marek
  Cc: linux-kernel, mcgrof, teg, kay, rusty, akpm, backports,
	Guenter Roeck, Greg Kroah-Hartman, rafael.j.wysocki,
	Dmitry Torokhov, Takashi Iwai, Christoph Hellwig,
	Mauro Carvalho Chehab, Johannes Berg, Hauke Mehrtens, Paul Bolle,
	Paul Gortmaker, Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:

> This patchset implements dynamic pegging of kconfig symbol
> into driver modinfo section

First a little bit of motivation here helps, so let me try to
help fill in some gaps. This may help explain what you have
been working on a bit more.

First, for those that were not Cc'd but curious about this work
so far, you can read the patches here:

Original cover letter:

https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com

https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5

There are a few situations in which you may want to extract a
subset of Kconfig CONFIG_* symbols for a system. At least when
only considering modules:

 a) When optimizing build requirements for a kernel for a system.
    That is you boot into a distro kernel and then want to build
    a slim kernel only with sensible kernel configuration options.

 b) When you are on a distribution kernel but the distribution
    kernel provided lacks hardware support for your device, you
    may either want to upgrade the full kernel in which case you
    want to do a) or -- you may want to just a backports release
    which provides just the modules you need, you'd use it on top
    of the distribution kernel.

Are there other *uses* outside of the two above for having a direct
deterministic mapping between module and a kconfig symbol ? What
if we could also do this for built-in code as well ? kconfig-sat [0] is
work that is still being ironed out, but if that was in place and we
handed it a set of required options we want, it should in theory help
resolve the dependencies to give us a functional .config for a kernel.
This of course is a long term possible goal just used as an example
-- but are there other uses?

Another possible gain here is if addressing a deterministic mapping might
help with some of the limitations observed in practice over years with
kconfig and semantics for keconfig. I go into some of this further below.

There are not many options currently available to optimize kernel builds this
way for modules or choose only what kernel kconfig options you need.  One of
the best known efforts is the upstream 'make localmodconfig' added by Steven
years ago to help with a). This has its own set of limitations though. For
backports we also need to address possible kconfig symbol name changes, and the
fact that you may boot a box without a driver loaded and as such not be sure
what new kernel kconfig symbol you may need.

There are shared issues with both goals, addressing shared issues
can ultimately help both. One of the shared known issues is the
lack of a deterministic resolution of a module to a respective
kconfig symbol. Even answering if this is possible or should be
possible requires some work.

The work Cristina has done here for this year's Google Summer of
Code is try to help start looking into addressing the question if
this is possible and then start looking into how we could annotate
such a thing, and later how we'd use it.

[0] https://kernelnewbies.org/KernelProjects/kconfig-sat

> * updates streamline_config.pl to generate the auxiliary file
> scripts/mod/Module.ksymb containing associations of driver file
> names and corresponding kconfig symbols CONFIG_*

Here you make use of existing heuristics -- note however that
heuristics are needed as we have no direct mapping available
nor do we have any rules over how to make such a direct mapping
possible if we wanted this mapping to be deterministic.

> * updates Makefiles to trigger streamline_config.pl call for
> Module.ksymb generation and pass the determined CONFIG_* symbol
> as compilation parameter -D in KBUILD_KSYMB variable

Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
to the build system a config symbol associated with a module, its the right
approach, however re-using streamline_config.pl seems questionable here, and it
seems an optimization might be / should be possible. Likewise we'd need to also
still look into and address *why* some modules don't end up with a respective
kconfig symbol -- and if we wanted to do something about this. Given the stated
goals more importantly would be what the costs are to have a deterministic
mapping.

> * adds kconfig_symb as module attribute

This seems lightweight and follows the logic of the other attributes.
If its too much, this can be a configurable feature.

> * updates modpost to set KBUILD_KSYMB macro as value for
> kbuild_symb attribute

Neat.

> Note: the content of the file Module.ksymb is generated for
> all modules that have only one associate CONFIG option. All
> others are considered to be components linked at the final
> modules but not final modules itselves.

To be clear, you only peg the CONFIG_ option if you are
100% certain of having a direct mapping, is that right ?

> The result of this patchset is the following. All modules from
> /sys expose the correct CONFIG_* symbol in the module attribute
> kconfig_symb with some exceptions. For a total number of 58
> modules, 

What kernel configuration did you use ? What architecture ?
If you use allmodconfig, what do the numbers look like ?

> 4 of them do not:
> 
> snd_seq_midi_event

Takashi, any reason for this to not have a Kconfig symbol ?
If we don't want one, is there perhaps any annotation we
could use to help make a deterministic association if a
respective kconfig symbol is needed ?

> mptscsih

Same thing -- for both I suppose we can infer that these
are just helpers, and can / should be enabled if any of
the devices are found to be present / required.

I can imagine that in certain situations a helper like
the above might really just be optional rather than required,
but I'd hope such an optional nature should be reflected via
Kconfig, the default setting being the recommended option to
assume as safe then, unless input is provided to help fill
in the gap and disable / force enable certain options.

> libahci

In this case libahci is spinkled over a lot of lines on the Makefile,
its a module though, and there is no direct association of dependency
relationship other than the Makefile association. Using a kconfig
symbol would replace the Makefile clutter but make the assocation
and requirement of the dependency explicit through Kconfig. Is
that desirable or should we just make the assumption that in these
cases we could assume one Kconfig symbol that adds a module implicitly
is equivalent as if the module had a kconfig symbol and we had done
"select foo" for each ? Note though that select is heavy handed [1] --
and its over use can lead to issues. For instance its known that
select does not check for dependencies, its a heavy handed way of
enabling options needed by architectures. So using select here may
not be desirable. Using "depends on" here would work but it can hide
from the menu drivers until you have enabled its dependency.

[1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e

> mptbase

Same situation as  mptscsih.

> After a short research:
> 
> For mptscsih - ./drivers/message/fusion/Makefile
> obj-$(CONFIG_FUSION_SPI)	+= mptbase.o mptscsih.o mptspi.o
> obj-$(CONFIG_FUSION_FC)		+= mptbase.o mptscsih.o mptfc.o
> obj-$(CONFIG_FUSION_SAS)	+= mptbase.o mptscsih.o mptsas.o
> As appears in the Makefile mptscsi is part of more config
> options so it's excluded because it's not considered a module
> by itself but, somehow gets compiled as module. Same for the
> mptbase. I still have to understand better the Kbuild logic so
> I cannot state that this is loose practice.

Another reason for this loose use could also be that using "select"
in Kconfig is heavy handed and can lead to recursive issues, refer
to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
dependency limitation / resolution") for more information and
a clear example of the issue. There is also a list of commits
with possible resolutions to issues, so that can be inspected further
to try to understand if perhaps some of the lack of kconfig symbols
and respective deterministic mapping from module to kconfig is due
to prior issues with select. I doubt it given that otherwise we'd
have at least symbols for these -- but worth taking into consideration
*if* we do then want to add symbols for them, we do not want to lead
to recursive issues if we add these new symbols.

If we add a kconfig symbol for modules lacking them, what is a
strategical way to do so in such a way we avoid known past
issues but also paves the way for the future ?

> For libahci there are multiple CONFIG_ symbols for different
> architectures.
> 
> Although past discussion recommended not to use streamline_config,
> this patchset still relies on it, for the sake of the proof of
> concept. 

OK.

> Although the result is close to the target it still does
> not provide complete correctness. However it can be replaced by
> creating another script

Or C code.

> which tries to determine better the
> module <-> CONFIG associations and output them in auxiliary file
> Module.ksymb. Maybe this way we could also determine all CONFIGs
> for a particular driver, not only the exact one that enables it.
> 
> The auxiliary file is necessary because the Makefile itself does
> not have a mapping. 

Clarification: at least not an easy O(1) quick mapping.

> The makefile includes the config file with
> directive include which creates a series of internal variables
> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
> other Makefiles, lines like 
> 
> obj-$(CONFIG_FOO) = foo.o 
> 
> are resolved in place to obj-y, obj-m etc according to 'make'
> logic and the association is lost.
> 
> Finally, what does this patchset provide is an infrastructure
> to dinamically peg CONFIG_* options to associate drivers using
> the mapping from Module.ksymb file. Generation of Module.ksymb
> can be replaced but keeping the same format permit the usage of
> the other patches.

OK this makes sense, so as it stands it seems we need a better
way to generate the Module.ksymb and then also address those
modules which lack a current deterministic mapping, and address
a resolution with the community on that.

> This patchset is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

Good stuff!

  Luis

> Cristina Moraru (5):
>   Add generation of Module.symb in streamline_config
>   Add CONFIG symbol to module as compilation parameter
>   Trigger Module.ksymb generation in Makefile
>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>   Add kconf_symb as kernel module attribute
> 
>  Makefile                             |  4 ++++
>  include/linux/module.h               |  1 +
>  kernel/module.c                      |  2 ++
>  scripts/Makefile.lib                 |  9 ++++++++-
>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>  scripts/mod/modpost.c                |  7 +++++++
>  6 files changed, 51 insertions(+), 2 deletions(-)
> 
> -- 
> 2.7.4
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter
       [not found] ` <1471462023-119645-3-git-send-email-cristina.moraru09@gmail.com>
@ 2016-08-18 18:10   ` Luis R. Rodriguez
  2016-08-18 18:55     ` Luis R. Rodriguez
  2016-08-20 15:11     ` Cristina-Gabriela Moraru
  0 siblings, 2 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 18:10 UTC (permalink / raw)
  To: Cristina Moraru; +Cc: linux-kernel, mcgrof, teg, kay, rusty, akpm, backports

On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> Add  CONFIG symbol to kernel modules as a define via -D

Perhaps better worded as:

When modules have a direct Kconfig CONFIG_ symbol associated with
we want to be able to make it available to the build system when we
are building the module.

You can then describe you do this with -D.

> compilation parameter. The CONFIG_FOO symbol for each
> module is determined by the module name, using the
> associations from Module.ksymb. This file is loaded
> using the 'include' directive, thus run like a regular
> makefile. The format of the content of the file is the
> following:
> 
> foo_KCONF=CONFIG_FOO
> 
> creating a set of Makefile variables foo_KCONF with the
> CONFIG_FOO as values.
> 
> The Makefile then adds this value in the compilation
> command with -DKBUILD_KCONF='"CONFIG_FOO"'.

Very nice. What would it mean if it lacks this ?
This should be explained on the commit log.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out as its part of the cover letter and
not relevant to the patch. You can however mention you will
use this later to make the define more useful in userspace.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/Makefile.lib | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index e7df0f5..8ae9b7f 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
>  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
>  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
>  
> +# Include Module.ksymb which contains the associations of modules' names
> +# and corresponding CONFIG_* options
> +include Module.ksymb

Is this file always generated? If not prefixing the include call with - would
be better. If we are going to make this optional, then an ifdef wrapper would
suffice as we know it'd be expected only if the new feature was enabled.

> +
>  # These flags are needed for modversions and compiling, so we define them here
>  # already
>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))

Are clashes possible with this formula? Can we end up with two results for instance?
If not what prevents current konfig logic and namespace from a clash ? If we do
not have anything to prevent a clash, what can we do to help make such clash not
possible ?

>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
> @@ -162,7 +169,7 @@ endif
>  
>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__c_flags) $(modkern_cflags)                           \
> -		 $(basename_flags) $(modname_flags)
> +		 $(basename_flags) $(modname_flags) $(ksymb_flags)
>  
>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__a_flags) $(modkern_aflags)
> -- 

Other than that looks good!

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config
       [not found] ` <1471462023-119645-2-git-send-email-cristina.moraru09@gmail.com>
@ 2016-08-18 18:22   ` Luis R. Rodriguez
  2016-08-18 18:32     ` Luis R. Rodriguez
       [not found]     ` <CAGZ2q2xi9Uy-ye387=mWhy_fOEJBC593Nos7fH027m-_ZdoOXA@mail.gmail.com>
  0 siblings, 2 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 18:22 UTC (permalink / raw)
  To: Cristina Moraru
  Cc: linux-kernel, mcgrof, teg, kay, rusty, akpm, backports, vegard.nossum

On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
> Add generation of ./scripts/Module.ksymb file containing
> associations of driver file names and corresponding CONFIG_*
> symbol:
> 
> foo_KCONF=CONFIG_FOO
> 
> This file will be further loaded by the Makefile as a
> configuration file: all foo_KCONF will be considered variables
> and all CONFIG_FOO will be their associate values.

Will the file generated always work? What are the current
restrictions on kconfig variable names?

> The suffix
> is added to be able to differentiate the new variables from
> all existing variables in the Makefile system. Each kernel
> module will receive its CONFIG_FOO option as macro in the
> compilation command (-D parameter) and will expose it along
> with other module attributes (attributes of struct module).

This is done later so this description does not belong here.
You want to describe what this patch does alone, if it has
potential for the future just mention basics, and that will will
be done later.

> Note: this patch is built on the assumption that each driver
> has exactly one associate CONFIG_FOO symbol.

This should note be a note, it should be an important aspect of
your changes -- you should try to describe when a direct mapping
is possible and when it is not.

> CONFIG_* options
> that are required by CONFIG_FOO are not included. They can be
> found by SAT resolvers.

You can leave this out.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> 
> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>  1 file changed, 29 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/kconfig/streamline_config.pl b/scripts/kconfig/streamline_config.pl
> index b8c7b29..686d02a 100755
> --- a/scripts/kconfig/streamline_config.pl
> +++ b/scripts/kconfig/streamline_config.pl
> @@ -128,9 +128,11 @@ my @config_file = read_config;
>  # Parse options
>  my $localmodconfig = 0;
>  my $localyesconfig = 0;
> +my $genmoduleksymb = 0;
>  
>  GetOptions("localmodconfig" => \$localmodconfig,
> -	   "localyesconfig" => \$localyesconfig);
> +	   "localyesconfig" => \$localyesconfig,
> +	   "genmoduleksymb" => \$genmoduleksymb);
>  
>  # Get the build source and top level Kconfig file (passed in)
>  my $ksource = ($ARGV[0] ? $ARGV[0] : '.');
> @@ -147,6 +149,7 @@ my %objects;
>  my $var;
>  my $iflevel = 0;
>  my @ifdeps;
> +my @drv_objs;
>  
>  # prevent recursion
>  my %read_kconfigs;
> @@ -348,6 +351,31 @@ foreach my $makefile (@makefiles) {
>      close($infile);
>  }
>  
> +foreach my $obj_key ( keys %objects )
> +{
> +	my @config_options = @{$objects{$obj_key}};
> +	# Last index of array is 0 is equivalent to array's size is 1
> +	if ( $#config_options == 0) {
> +		push(@drv_objs, $obj_key);
> +	}
> +}
> +
> +sub gen_module_kconfigs {
> +	my $module_ksymb = $ENV{'srctree'}."/scripts/Module.ksymb";
> +
> +	open(my $ksymbfile, '>', $module_ksymb) || die "Can not open $module_ksymb for writing";
> +
> +	foreach (@drv_objs) {
> +		print $ksymbfile "$_" . "_KCONF=" . "@{$objects{$_}}\n";
> +	}
> +	close($ksymbfile);
> +}
> +
> +if ($genmoduleksymb) {
> +	gen_module_kconfigs();
> +	exit(0);
> +}
> +
>  my %modules;
>  my $linfile;

This is nice for experimentation, however as you note we want
to evaluate a better way to do this. While at it, can you describe
what algorithm currently is used by scripts/kconfig/streamline_config.pl
to collect %objects and the associated CONFIG symbol ? Can you think
of possible flaws to it ?

The generation here is also forced, we want to enable this to be optional
so please consider adding a Kconfig entry for this as a feature, just as
the module versioning stuff has one.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile
       [not found] ` <1471462023-119645-4-git-send-email-cristina.moraru09@gmail.com>
@ 2016-08-18 18:30   ` Luis R. Rodriguez
  0 siblings, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 18:30 UTC (permalink / raw)
  To: Cristina Moraru
  Cc: linux-kernel, mcgrof, teg, kay, rusty, akpm, Fengguang Wu,
	backports, vegard.nossum

On Wed, Aug 17, 2016 at 09:27:01PM +0200, Cristina Moraru wrote:
> Trigger the generation of file scripts/Module.ksymb in
> Makefile by calling updated scripts/streamline_config.pl
> with corresponding parameter (--genmodulesymb). The file
> is generated at each compilation considering that
> associations may change after tree updates.

I really like this approach as you make the resolver
configurable, this lets others experiment and consider
alternatives.

make KSYMB_GENERATOR=/path/research/my-config-generator-is-better

The fact that you can do this should be documented as part of
the patch commit log.

You'll want to wrap this generation under a CONFIG option so only
kernels that want generate this file.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> 
> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 393b615..286b949 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -219,6 +219,10 @@ VPATH		:= $(srctree)$(if $(KBUILD_EXTMOD),:$(KBUILD_EXTMOD))
>  
>  export srctree objtree VPATH
>  
> +KSYMB_GENERATOR := $(objtree)/scripts/kconfig/streamline_config.pl
> +ksymb_gen_command = perl $(KSYMB_GENERATOR) --genmoduleksymb $(objtree) $(Kconfig)

$(PERL)

> +ksymb_update := $(shell objtree=$(objtree) srctree=$(srctree) $(ksymb_gen_command))
> +
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
>  # line overrides the setting of ARCH below.  If a native build is happening,
> -- 
> 2.7.4

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config
  2016-08-18 18:22   ` [RFC PATCH 1/5] Add generation of Module.symb in streamline_config Luis R. Rodriguez
@ 2016-08-18 18:32     ` Luis R. Rodriguez
       [not found]     ` <CAGZ2q2xi9Uy-ye387=mWhy_fOEJBC593Nos7fH027m-_ZdoOXA@mail.gmail.com>
  1 sibling, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 18:32 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Cristina Moraru, linux-kernel, teg, kay, rusty, akpm, backports,
	vegard.nossum


Also I think your subject has a typo.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter
  2016-08-18 18:10   ` [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter Luis R. Rodriguez
@ 2016-08-18 18:55     ` Luis R. Rodriguez
  2016-08-20 15:11     ` Cristina-Gabriela Moraru
  1 sibling, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 18:55 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Cristina Moraru, linux-kernel, teg, kay, rusty, akpm, backports,
	vegard.nossum, Fengguang Wu, Guenter Roeck, Steven Rostedt

On Thu, Aug 18, 2016 at 08:10:07PM +0200, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> 
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/Makefile.lib | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> > index e7df0f5..8ae9b7f 100644
> > --- a/scripts/Makefile.lib
> > +++ b/scripts/Makefile.lib
> > @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
> >  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
> >  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
> >  
> > +# Include Module.ksymb which contains the associations of modules' names
> > +# and corresponding CONFIG_* options
> > +include Module.ksymb
> 
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

The ordering of the patches 2 and 3 seem to be backward, you should first
generate the file otherwise this patch will break the build as the file
is not present.

> > +
> >  # These flags are needed for modversions and compiling, so we define them here
> >  # already
> >  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> > @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> > +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> > +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> > +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> 
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

To help test this I've added your patches to a development branch on my linux-next
tree to see if 0-day picks up any issues.

Guenter, to be even more thorough can I trouble you for a spin of this branch?

https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160818-gsoc-kconf_symb

Results should help give us an idea of troubling areas if this were to be added
as a new Kconfig feature even for R&D.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
       [not found] ` <1471462023-119645-5-git-send-email-cristina.moraru09@gmail.com>
@ 2016-08-18 18:59   ` Luis R. Rodriguez
  2016-08-20 15:16     ` Cristina-Gabriela Moraru
  0 siblings, 1 reply; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 18:59 UTC (permalink / raw)
  To: Cristina Moraru
  Cc: linux-kernel, mcgrof, teg, kay, rusty, akpm, Steven Rostedt,
	backports, vegard.nossum

On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> Update modpost to add in *.mod.c files generated for each
> module the setting of module attribute kernel_ksymb to
> value given by KBUILD_KSYMB macro.

Please review your patches and update subjects to match what other
types of previous patches look like in terms of format, so
for this patch check changes to scripts/mod/modpost.c and use
similar type of patch prefix.

> 
> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/mod/modpost.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> index 48958d3..a105916 100644
> --- a/scripts/mod/modpost.c
> +++ b/scripts/mod/modpost.c
> @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
>  	}
>  }
>  
> +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> +{
> +	buf_printf(b, "\n");
> +	buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");

What if its not available? What happens?

> +}
> +
>  static void write_if_changed(struct buffer *b, const char *fname)
>  {
>  	char *tmp;
> @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
>  		add_depends(&buf, mod, modules);
>  		add_moddevtable(&buf, mod);
>  		add_srcversion(&buf, mod);
> +		add_kconfig_symbol(&buf, mod);
>  
>  		sprintf(fname, "%s.mod.c", mod->name);
>  		write_if_changed(&buf, fname);
> -- 
> 2.7.4
> 
> 

-- 
Luis Rodriguez, SUSE LINUX GmbH
Maxfeldstrasse 5; D-90409 Nuernberg
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 5/5] Add kconf_symb as kernel module attribute
       [not found] ` <1471462023-119645-6-git-send-email-cristina.moraru09@gmail.com>
@ 2016-08-18 19:02   ` " Luis R. Rodriguez
  0 siblings, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-18 19:02 UTC (permalink / raw)
  To: Cristina Moraru
  Cc: linux-kernel, mcgrof, teg, kay, rusty, akpm, vegard.nossum,
	Steven Rostedt, backports

On Wed, Aug 17, 2016 at 09:27:03PM +0200, Cristina Moraru wrote:
> Add kconf_symbol attribute to kernel struct module.

Nice, you can describe here what and how this can be
used by userspace, for instance you can mention how
udevadm might use this, etc.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  include/linux/module.h | 1 +
>  kernel/module.c        | 2 ++
>  2 files changed, 3 insertions(+)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3daf2b3..bef5e44 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -353,6 +353,7 @@ struct module {
>  	struct module_attribute *modinfo_attrs;
>  	const char *version;
>  	const char *srcversion;
> +	const char *kconfig_symbol;
>  	struct kobject *holders_dir;
>  
>  	/* Exported symbols */
> diff --git a/kernel/module.c b/kernel/module.c
> index 5f71aa6..bb27300 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -757,6 +757,7 @@ static struct module_attribute modinfo_##field = {                    \
>  
>  MODINFO_ATTR(version);
>  MODINFO_ATTR(srcversion);
> +MODINFO_ATTR(kconfig_symbol);
>  
>  static char last_unloaded_module[MODULE_NAME_LEN+1];
>  
> @@ -1229,6 +1230,7 @@ static struct module_attribute *modinfo_attrs[] = {
>  	&module_uevent,
>  	&modinfo_version,
>  	&modinfo_srcversion,
> +	&modinfo_kconfig_symbol,
>  	&modinfo_initstate,
>  	&modinfo_coresize,
>  	&modinfo_initsize,
> -- 

Do we #ifdef over other features that are optional?

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-18 17:55 ` [RFC PATCH 0/5] Add CONFIG symbol as module attribute Luis R. Rodriguez
@ 2016-08-19  9:07   ` Michal Marek
  2016-08-22 19:48     ` Cristina-Gabriela Moraru
  2016-08-23 21:32     ` Luis R. Rodriguez
  2016-08-22 19:35   ` Cristina-Gabriela Moraru
  2016-08-25  7:43   ` Christoph Hellwig
  2 siblings, 2 replies; 28+ messages in thread
From: Michal Marek @ 2016-08-19  9:07 UTC (permalink / raw)
  To: Luis R. Rodriguez, Cristina Moraru, vegard.nossum,
	Valentin Rothberg, Hannes Reinecke, Sam Ravnborg
  Cc: linux-kernel, teg, kay, rusty, akpm, backports, Guenter Roeck,
	Greg Kroah-Hartman, rafael.j.wysocki, Dmitry Torokhov,
	Takashi Iwai, Christoph Hellwig, Mauro Carvalho Chehab,
	Johannes Berg, Hauke Mehrtens, Paul Bolle, Paul Gortmaker,
	Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> 
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
> 
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
> 
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
> 
> Original cover letter:
> 
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> 
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> 
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
> 
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
> 
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.

c) Having the mapping in sysfs would allow to simplify
streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
did not depend on streamline_config.pl :). One idea would be to generate
the Module.ksymb in a similar way we generate the modules.builtin file:
Generate an alternate include/config/*.conf with all CONFIG_FOO=m
replaced with

CONFIG_FOO=m-CONFIG_FOO

and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
create the mapping. This would also properly cover cases where we build
the $(obj-m) list from another list. It would certainly create other
corner cases, but it's worth trying IMO.

Another thing is that we do not necessarily need to record this
information in .modinfo, but we can generate a list in
/lib/modules/`uname -r`/ for consumption. It could also include drivers
that are builtin in the current configuration.

Michal
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config
       [not found]     ` <CAGZ2q2xi9Uy-ye387=mWhy_fOEJBC593Nos7fH027m-_ZdoOXA@mail.gmail.com>
@ 2016-08-20 14:49       ` Cristina-Gabriela Moraru
  2016-08-23 19:00       ` Luis R. Rodriguez
  1 sibling, 0 replies; 28+ messages in thread
From: Cristina-Gabriela Moraru @ 2016-08-20 14:49 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: linux-kernel, Tom Gundersen, Kay Sievers, Rusty Russell, akpm,
	backports, vegard.nossum

2016-08-20 15:59 GMT+02:00 Cristina-Gabriela Moraru
<cristina.moraru09@gmail.com>:
>
>
> 2016-08-18 20:22 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
>>
>> On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
>> > Add generation of ./scripts/Module.ksymb file containing
>> > associations of driver file names and corresponding CONFIG_*
>> > symbol:
>> >
>> > foo_KCONF=3DCONFIG_FOO
>> >
>> > This file will be further loaded by the Makefile as a
>> > configuration file: all foo_KCONF will be considered variables
>> > and all CONFIG_FOO will be their associate values.
>>
>> Will the file generated always work? What are the current
>> restrictions on kconfig variable names?
>>
>
> The file generated will always work. It's loaded with makefile's "include=
"
> directive which is quite straightforward.
>
> If you're referring to  CONFIG_* variables, I can't find any restriction
> regarding size or anything else.
> If you're referring to the newly created makefile variable foo_KCONF I on=
ly
> found this in the documentation:
> "A variable name may be any sequence of characters not containing =E2=80=
=98:=E2=80=99, =E2=80=98#=E2=80=99,
> =E2=80=98=3D=E2=80=99, or whitespace."
> "It is traditional to use upper case letters in variable names, but we
> recommend using lower case letters for variable names that serve internal
> purposes in the makefile, and reserving upper case for parameters that
> control implicit rules or for parameters that the user should override wi=
th
> command options"
>
> Since the module name is lowercase I put the suffix uppercase in order to=
 be
> more visible.
>
>>
>> > The suffix
>> > is added to be able to differentiate the new variables from
>> > all existing variables in the Makefile system. Each kernel
>> > module will receive its CONFIG_FOO option as macro in the
>> > compilation command (-D parameter) and will expose it along
>> > with other module attributes (attributes of struct module).
>>
>> This is done later so this description does not belong here.
>> You want to describe what this patch does alone, if it has
>> potential for the future just mention basics, and that will will
>> be done later.
>>
>> > Note: this patch is built on the assumption that each driver
>> > has exactly one associate CONFIG_FOO symbol.
>>
>> This should note be a note, it should be an important aspect of
>> your changes -- you should try to describe when a direct mapping
>> is possible and when it is not.
>>
>> > CONFIG_* options
>> > that are required by CONFIG_FOO are not included. They can be
>> > found by SAT resolvers.
>>
>> You can leave this out.
>>
>> > This patch is part of a research project within
>> > Google Summer of Code of porting 'make localmodconfig'
>> > for backported drivers. The goal is to enable each
>> > module to expose in /sys its corresponding CONFIG_* option.
>> > The value of this attribute will be dynamically pegged by
>> > modpost without requiring extra work from the driver developers.
>> > Further, this information will be used by a hardware interogation
>> > tool to extract build information about the existing devices.
>>
>> You can leave this out.
>>
>> >
>> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
>> > ---
>> >  scripts/kconfig/streamline_config.pl | 30
>> > +++++++++++++++++++++++++++++-
>> >  1 file changed, 29 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/scripts/kconfig/streamline_config.pl
>> > b/scripts/kconfig/streamline_config.pl
>> > index b8c7b29..686d02a 100755
>> > --- a/scripts/kconfig/streamline_config.pl
>> > +++ b/scripts/kconfig/streamline_config.pl
>> > @@ -128,9 +128,11 @@ my @config_file =3D read_config;
>> >  # Parse options
>> >  my $localmodconfig =3D 0;
>> >  my $localyesconfig =3D 0;
>> > +my $genmoduleksymb =3D 0;
>> >
>> >  GetOptions("localmodconfig" =3D> \$localmodconfig,
>> > -        "localyesconfig" =3D> \$localyesconfig);
>> > +        "localyesconfig" =3D> \$localyesconfig,
>> > +        "genmoduleksymb" =3D> \$genmoduleksymb);
>> >
>> >  # Get the build source and top level Kconfig file (passed in)
>> >  my $ksource =3D ($ARGV[0] ? $ARGV[0] : '.');
>> > @@ -147,6 +149,7 @@ my %objects;
>> >  my $var;
>> >  my $iflevel =3D 0;
>> >  my @ifdeps;
>> > +my @drv_objs;
>> >
>> >  # prevent recursion
>> >  my %read_kconfigs;
>> > @@ -348,6 +351,31 @@ foreach my $makefile (@makefiles) {
>> >      close($infile);
>> >  }
>> >
>> > +foreach my $obj_key ( keys %objects )
>> > +{
>> > +     my @config_options =3D @{$objects{$obj_key}};
>> > +     # Last index of array is 0 is equivalent to array's size is 1
>> > +     if ( $#config_options =3D=3D 0) {
>> > +             push(@drv_objs, $obj_key);
>> > +     }
>> > +}
>> > +
>> > +sub gen_module_kconfigs {
>> > +     my $module_ksymb =3D $ENV{'srctree'}."/scripts/Module.ksymb";
>> > +
>> > +     open(my $ksymbfile, '>', $module_ksymb) || die "Can not open
>> > $module_ksymb for writing";
>> > +
>> > +     foreach (@drv_objs) {
>> > +             print $ksymbfile "$_" . "_KCONF=3D" . "@{$objects{$_}}\n=
";
>> > +     }
>> > +     close($ksymbfile);
>> > +}
>> > +
>> > +if ($genmoduleksymb) {
>> > +     gen_module_kconfigs();
>> > +     exit(0);
>> > +}
>> > +
>> >  my %modules;
>> >  my $linfile;
>>
>> This is nice for experimentation, however as you note we want
>> to evaluate a better way to do this. While at it, can you describe
>> what algorithm currently is used by scripts/kconfig/streamline_config.pl
>> to collect %objects and the associated CONFIG symbol ? Can you think
>> of possible flaws to it ?
>>
>
> Shortly, all makefiles are parsed searching for patterns such as:
>
> obj-$(CONFIG_FOO) :=3D obj-foo1.o obj-foo2.o ... etc.
>
> Each obj-foo is added in the hashmap as key with the value
> array(CONFIG_FOO).
> If obj-foo already exists in the hashmap CONFIG_FOO is appended to the ar=
ray
> value.
>
> I noted that there are situations where an object obj-foo has as associat=
es
> more identical CONFIG_FOOs.
> Since I select only the names which have exaclty one CONFIG_* associated
> this is a bit confusing. I am missing one category of drivers which can b=
e
> found:
>
> driver-foo ---- CONFIG_FOO CONFIG_FOO
>
> For example: nfit CONFIG_ACPI_NFIT CONFIG_ACPI_NFIT -- this actually resu=
lts
> in  a module in /sys which does not have a content into kconfig_ksymb
> although it has exactly one match.
>
> Another thing I can think of is that sometimes we find lines such as:
>
> foo-name-$(CONFIG_FOO) =3D obj-foo1.o obj-foo2.o obj-foo3.o
>
> This can be a clue that the final kernel module is named foo-name and cou=
ld
> solve some ambiguous cases.
> This are just a few situations I can think of to be kept in mind for the
> next Module.ksymb generator
>
>>
>> The generation here is also forced, we want to enable this to be optiona=
l
>> so please consider adding a Kconfig entry for this as a feature, just as
>> the module versioning stuff has one.
>
>
> OK
>>
>>
>>   Luis
>
>

Message forwarded since original message was rejected by lkml and
backports for being HTML format.
Cristina
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter
  2016-08-18 18:10   ` [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter Luis R. Rodriguez
  2016-08-18 18:55     ` Luis R. Rodriguez
@ 2016-08-20 15:11     ` Cristina-Gabriela Moraru
  2016-08-23 19:07       ` Luis R. Rodriguez
  1 sibling, 1 reply; 28+ messages in thread
From: Cristina-Gabriela Moraru @ 2016-08-20 15:11 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: linux-kernel, Tom Gundersen, Kay Sievers, Rusty Russell, akpm, backports

2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
>> Add  CONFIG symbol to kernel modules as a define via -D
>
> Perhaps better worded as:
>
> When modules have a direct Kconfig CONFIG_ symbol associated with
> we want to be able to make it available to the build system when we
> are building the module.
>
> You can then describe you do this with -D.
>
>> compilation parameter. The CONFIG_FOO symbol for each
>> module is determined by the module name, using the
>> associations from Module.ksymb. This file is loaded
>> using the 'include' directive, thus run like a regular
>> makefile. The format of the content of the file is the
>> following:
>>
>> foo_KCONF=CONFIG_FOO
>>
>> creating a set of Makefile variables foo_KCONF with the
>> CONFIG_FOO as values.
>>
>> The Makefile then adds this value in the compilation
>> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
>
> Very nice. What would it mean if it lacks this ?
> This should be explained on the commit log.
>

If we lack it then KBUILD_KCONF="" and in /sys the module has the
kconfig_symbol attribute but with the empty string as content:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

I will add it into the commit log.

>> This patch is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> You can leave this out as its part of the cover letter and
> not relevant to the patch. You can however mention you will
> use this later to make the define more useful in userspace.
>
>> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
>> ---
>>  scripts/Makefile.lib | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
>> index e7df0f5..8ae9b7f 100644
>> --- a/scripts/Makefile.lib
>> +++ b/scripts/Makefile.lib
>> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
>>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
>>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
>>
>> +# Include Module.ksymb which contains the associations of modules' names
>> +# and corresponding CONFIG_* options
>> +include Module.ksymb
>
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

Yes. This file is always generated. In case a 'git pull' happened
between two 'make' commands, the associations should be updated. Ok. I
will add a ifdef.

>
>> +
>>  # These flags are needed for modversions and compiling, so we define them here
>>  # already
>>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
>> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
>> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
>> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
>> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
>
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

I think the fact that modname is unique prevents from having clashes.
KBUILD_KSYMB is found according to modname.
Currently there is no clash because I enforced the Module.ksymb to
have 1-to-1 mapping.
The only potential clash I can imagine right now it having the same
module name but architecture specific CONFIG_* symbol. If there is a
1-to-1 mapping in Module.ksymb there should be no clash in the
makefile.

>
>>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
>> @@ -162,7 +169,7 @@ endif
>>
>>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__c_flags) $(modkern_cflags)                           \
>> -              $(basename_flags) $(modname_flags)
>> +              $(basename_flags) $(modname_flags) $(ksymb_flags)
>>
>>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__a_flags) $(modkern_aflags)
>> --
>
> Other than that looks good!
>
>   Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
  2016-08-18 18:59   ` [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute Luis R. Rodriguez
@ 2016-08-20 15:16     ` Cristina-Gabriela Moraru
  2016-08-23 19:10       ` Luis R. Rodriguez
  0 siblings, 1 reply; 28+ messages in thread
From: Cristina-Gabriela Moraru @ 2016-08-20 15:16 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: linux-kernel, Tom Gundersen, Kay Sievers, Rusty Russell, akpm,
	Steven Rostedt, backports, vegard.nossum

2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
>
> On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > Update modpost to add in *.mod.c files generated for each
> > module the setting of module attribute kernel_ksymb to
> > value given by KBUILD_KSYMB macro.
>
> Please review your patches and update subjects to match what other
> types of previous patches look like in terms of format, so
> for this patch check changes to scripts/mod/modpost.c and use
> similar type of patch prefix.
>

OK

>
> >
> > This patch is part of a research project within
> > Google Summer of Code of porting 'make localmodconfig'
> > for backported drivers. The goal is to enable each
> > module to expose in /sys its corresponding CONFIG_* option.
> > The value of this attribute will be dynamically pegged by
> > modpost without requiring extra work from the driver developers.
> > Further, this information will be used by a hardware interogation
> > tool to extract build information about the existing devices.
>
> You can leave this out.
>
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/mod/modpost.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > index 48958d3..a105916 100644
> > --- a/scripts/mod/modpost.c
> > +++ b/scripts/mod/modpost.c
> > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> >       }
> >  }
> >
> > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > +{
> > +     buf_printf(b, "\n");
> > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
>
> What if its not available? What happens?
>

If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
More concrete:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

>
> > +}
> > +
> >  static void write_if_changed(struct buffer *b, const char *fname)
> >  {
> >       char *tmp;
> > @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
> >               add_depends(&buf, mod, modules);
> >               add_moddevtable(&buf, mod);
> >               add_srcversion(&buf, mod);
> > +             add_kconfig_symbol(&buf, mod);
> >
> >               sprintf(fname, "%s.mod.c", mod->name);
> >               write_if_changed(&buf, fname);
> > --
> > 2.7.4
> >
> >
>
> --
> Luis Rodriguez, SUSE LINUX GmbH
> Maxfeldstrasse 5; D-90409 Nuernberg
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-18 17:55 ` [RFC PATCH 0/5] Add CONFIG symbol as module attribute Luis R. Rodriguez
  2016-08-19  9:07   ` Michal Marek
@ 2016-08-22 19:35   ` Cristina-Gabriela Moraru
  2016-08-23 19:17     ` Luis R. Rodriguez
  2016-08-25  7:43   ` Christoph Hellwig
  2 siblings, 1 reply; 28+ messages in thread
From: Cristina-Gabriela Moraru @ 2016-08-22 19:35 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: vegard.nossum, Valentin Rothberg, Hannes Reinecke, Sam Ravnborg,
	Michal Marek, linux-kernel, Tom Gundersen, Kay Sievers,
	Rusty Russell, akpm, backports, Guenter Roeck,
	Greg Kroah-Hartman, rafael.j.wysocki, Dmitry Torokhov,
	Takashi Iwai, Christoph Hellwig, Mauro Carvalho Chehab,
	Johannes Berg, Hauke Mehrtens, Paul Bolle, Paul Gortmaker,
	Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
>
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
>
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
>
> Original cover letter:
>
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
>
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
>
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.
>
> Are there other *uses* outside of the two above for having a direct
> deterministic mapping between module and a kconfig symbol ? What
> if we could also do this for built-in code as well ? kconfig-sat [0] is
> work that is still being ironed out, but if that was in place and we
> handed it a set of required options we want, it should in theory help
> resolve the dependencies to give us a functional .config for a kernel.
> This of course is a long term possible goal just used as an example
> -- but are there other uses?
>
I can't think of any examples. I will add this ones in the cover
letter as motivation.

> Another possible gain here is if addressing a deterministic mapping might
> help with some of the limitations observed in practice over years with
> kconfig and semantics for keconfig. I go into some of this further below.
>
> There are not many options currently available to optimize kernel builds this
> way for modules or choose only what kernel kconfig options you need.  One of
> the best known efforts is the upstream 'make localmodconfig' added by Steven
> years ago to help with a). This has its own set of limitations though. For
> backports we also need to address possible kconfig symbol name changes, and the
> fact that you may boot a box without a driver loaded and as such not be sure
> what new kernel kconfig symbol you may need.
>
> There are shared issues with both goals, addressing shared issues
> can ultimately help both. One of the shared known issues is the
> lack of a deterministic resolution of a module to a respective
> kconfig symbol. Even answering if this is possible or should be
> possible requires some work.
>
> The work Cristina has done here for this year's Google Summer of
> Code is try to help start looking into addressing the question if
> this is possible and then start looking into how we could annotate
> such a thing, and later how we'd use it.
>
> [0] https://kernelnewbies.org/KernelProjects/kconfig-sat
>
>> * updates streamline_config.pl to generate the auxiliary file
>> scripts/mod/Module.ksymb containing associations of driver file
>> names and corresponding kconfig symbols CONFIG_*
>
> Here you make use of existing heuristics -- note however that
> heuristics are needed as we have no direct mapping available
> nor do we have any rules over how to make such a direct mapping
> possible if we wanted this mapping to be deterministic.
>
>> * updates Makefiles to trigger streamline_config.pl call for
>> Module.ksymb generation and pass the determined CONFIG_* symbol
>> as compilation parameter -D in KBUILD_KSYMB variable
>
> Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
> to the build system a config symbol associated with a module, its the right
> approach, however re-using streamline_config.pl seems questionable here, and it
> seems an optimization might be / should be possible. Likewise we'd need to also
> still look into and address *why* some modules don't end up with a respective
> kconfig symbol -- and if we wanted to do something about this. Given the stated
> goals more importantly would be what the costs are to have a deterministic
> mapping.
>

Yes. I've noticed. I will change it completely.

>> * adds kconfig_symb as module attribute
>
> This seems lightweight and follows the logic of the other attributes.
> If its too much, this can be a configurable feature.
>
>> * updates modpost to set KBUILD_KSYMB macro as value for
>> kbuild_symb attribute
>
> Neat.
>
>> Note: the content of the file Module.ksymb is generated for
>> all modules that have only one associate CONFIG option. All
>> others are considered to be components linked at the final
>> modules but not final modules itselves.
>
> To be clear, you only peg the CONFIG_ option if you are
> 100% certain of having a direct mapping, is that right ?
>

Yes. And there are some drivers which have duplicate values in the hash.
foo.o -> CONFIG_FOO CONFIG_FOO

Although this one has one match it's filtered as having 2 CONFIGs.

>> The result of this patchset is the following. All modules from
>> /sys expose the correct CONFIG_* symbol in the module attribute
>> kconfig_symb with some exceptions. For a total number of 58
>> modules,
>
> What kernel configuration did you use ? What architecture ?
> If you use allmodconfig, what do the numbers look like ?
>

I have a virtual machine in which I created the configuration with
make localmodconfig and installed the new kernel.
$  uname -a
Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
x86_64 x86_64 GNU/Linux

Should I send you the .config file ?

>> 4 of them do not:
>>
>> snd_seq_midi_event
>
> Takashi, any reason for this to not have a Kconfig symbol ?
> If we don't want one, is there perhaps any annotation we
> could use to help make a deterministic association if a
> respective kconfig symbol is needed ?
>
>> mptscsih
>
> Same thing -- for both I suppose we can infer that these
> are just helpers, and can / should be enabled if any of
> the devices are found to be present / required.
>
> I can imagine that in certain situations a helper like
> the above might really just be optional rather than required,
> but I'd hope such an optional nature should be reflected via
> Kconfig, the default setting being the recommended option to
> assume as safe then, unless input is provided to help fill
> in the gap and disable / force enable certain options.
>
>> libahci
>
> In this case libahci is spinkled over a lot of lines on the Makefile,
> its a module though, and there is no direct association of dependency
> relationship other than the Makefile association. Using a kconfig
> symbol would replace the Makefile clutter but make the assocation
> and requirement of the dependency explicit through Kconfig. Is
> that desirable or should we just make the assumption that in these
> cases we could assume one Kconfig symbol that adds a module implicitly
> is equivalent as if the module had a kconfig symbol and we had done
> "select foo" for each ? Note though that select is heavy handed [1] --
> and its over use can lead to issues. For instance its known that
> select does not check for dependencies, its a heavy handed way of
> enabling options needed by architectures. So using select here may
> not be desirable. Using "depends on" here would work but it can hide
> from the menu drivers until you have enabled its dependency.
>
> [1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e
>
>> mptbase
>
> Same situation as  mptscsih.
>
>> After a short research:
>>
>> For mptscsih - ./drivers/message/fusion/Makefile
>> obj-$(CONFIG_FUSION_SPI)      += mptbase.o mptscsih.o mptspi.o
>> obj-$(CONFIG_FUSION_FC)               += mptbase.o mptscsih.o mptfc.o
>> obj-$(CONFIG_FUSION_SAS)      += mptbase.o mptscsih.o mptsas.o
>> As appears in the Makefile mptscsi is part of more config
>> options so it's excluded because it's not considered a module
>> by itself but, somehow gets compiled as module. Same for the
>> mptbase. I still have to understand better the Kbuild logic so
>> I cannot state that this is loose practice.
>
> Another reason for this loose use could also be that using "select"
> in Kconfig is heavy handed and can lead to recursive issues, refer
> to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
> dependency limitation / resolution") for more information and
> a clear example of the issue. There is also a list of commits
> with possible resolutions to issues, so that can be inspected further
> to try to understand if perhaps some of the lack of kconfig symbols
> and respective deterministic mapping from module to kconfig is due
> to prior issues with select. I doubt it given that otherwise we'd
> have at least symbols for these -- but worth taking into consideration
> *if* we do then want to add symbols for them, we do not want to lead
> to recursive issues if we add these new symbols.
>
> If we add a kconfig symbol for modules lacking them, what is a
> strategical way to do so in such a way we avoid known past
> issues but also paves the way for the future ?
>
>> For libahci there are multiple CONFIG_ symbols for different
>> architectures.
>>
>> Although past discussion recommended not to use streamline_config,
>> this patchset still relies on it, for the sake of the proof of
>> concept.
>
> OK.
>
>> Although the result is close to the target it still does
>> not provide complete correctness. However it can be replaced by
>> creating another script
>
> Or C code.
>
>> which tries to determine better the
>> module <-> CONFIG associations and output them in auxiliary file
>> Module.ksymb. Maybe this way we could also determine all CONFIGs
>> for a particular driver, not only the exact one that enables it.
>>
>> The auxiliary file is necessary because the Makefile itself does
>> not have a mapping.
>
> Clarification: at least not an easy O(1) quick mapping.
>
>> The makefile includes the config file with
>> directive include which creates a series of internal variables
>> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
>> other Makefiles, lines like
>>
>> obj-$(CONFIG_FOO) = foo.o
>>
>> are resolved in place to obj-y, obj-m etc according to 'make'
>> logic and the association is lost.
>>
>> Finally, what does this patchset provide is an infrastructure
>> to dinamically peg CONFIG_* options to associate drivers using
>> the mapping from Module.ksymb file. Generation of Module.ksymb
>> can be replaced but keeping the same format permit the usage of
>> the other patches.
>
> OK this makes sense, so as it stands it seems we need a better
> way to generate the Module.ksymb and then also address those
> modules which lack a current deterministic mapping, and address
> a resolution with the community on that.
>
>> This patchset is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> Good stuff!
>
>   Luis
>
>> Cristina Moraru (5):
>>   Add generation of Module.symb in streamline_config
>>   Add CONFIG symbol to module as compilation parameter
>>   Trigger Module.ksymb generation in Makefile
>>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>>   Add kconf_symb as kernel module attribute
>>
>>  Makefile                             |  4 ++++
>>  include/linux/module.h               |  1 +
>>  kernel/module.c                      |  2 ++
>>  scripts/Makefile.lib                 |  9 ++++++++-
>>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>>  scripts/mod/modpost.c                |  7 +++++++
>>  6 files changed, 51 insertions(+), 2 deletions(-)
>>
>> --
>> 2.7.4
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-19  9:07   ` Michal Marek
@ 2016-08-22 19:48     ` Cristina-Gabriela Moraru
  2016-08-23 21:32     ` Luis R. Rodriguez
  1 sibling, 0 replies; 28+ messages in thread
From: Cristina-Gabriela Moraru @ 2016-08-22 19:48 UTC (permalink / raw)
  To: Michal Marek
  Cc: Luis R. Rodriguez, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, linux-kernel, Tom Gundersen,
	Kay Sievers, Rusty Russell, akpm, backports, Guenter Roeck,
	Greg Kroah-Hartman, rafael.j.wysocki, Dmitry Torokhov,
	Takashi Iwai, Christoph Hellwig, Mauro Carvalho Chehab,
	Johannes Berg, Hauke Mehrtens, Paul Bolle, Paul Gortmaker,
	Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

2016-08-19 11:07 GMT+02:00 Michal Marek <mmarek@suse.com>:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>
>>> This patchset implements dynamic pegging of kconfig symbol
>>> into driver modinfo section
>>
>> First a little bit of motivation here helps, so let me try to
>> help fill in some gaps. This may help explain what you have
>> been working on a bit more.
>>
>> First, for those that were not Cc'd but curious about this work
>> so far, you can read the patches here:
>>
>> Original cover letter:
>>
>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>
>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>
>> There are a few situations in which you may want to extract a
>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> only considering modules:
>>
>>  a) When optimizing build requirements for a kernel for a system.
>>     That is you boot into a distro kernel and then want to build
>>     a slim kernel only with sensible kernel configuration options.
>>
>>  b) When you are on a distribution kernel but the distribution
>>     kernel provided lacks hardware support for your device, you
>>     may either want to upgrade the full kernel in which case you
>>     want to do a) or -- you may want to just a backports release
>>     which provides just the modules you need, you'd use it on top
>>     of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
> did not depend on streamline_config.pl :). One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
>
> CONFIG_FOO=m-CONFIG_FOO
>
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.
>

That's interesting.

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.
>
> Michal

Thank you,

Cristina
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config
       [not found]     ` <CAGZ2q2xi9Uy-ye387=mWhy_fOEJBC593Nos7fH027m-_ZdoOXA@mail.gmail.com>
  2016-08-20 14:49       ` Cristina-Gabriela Moraru
@ 2016-08-23 19:00       ` Luis R. Rodriguez
  1 sibling, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-23 19:00 UTC (permalink / raw)
  To: Cristina-Gabriela Moraru
  Cc: Luis R. Rodriguez, linux-kernel, Tom Gundersen, Kay Sievers,
	Rusty Russell, akpm, backports, vegard.nossum

On Sat, Aug 20, 2016 at 03:59:17PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:22 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> 
> > On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
> > > Add generation of ./scripts/Module.ksymb file containing
> > > associations of driver file names and corresponding CONFIG_*
> > > symbol:
> > >
> > > foo_KCONF=CONFIG_FOO
> > >
> > > This file will be further loaded by the Makefile as a
> > > configuration file: all foo_KCONF will be considered variables
> > > and all CONFIG_FOO will be their associate values.
> >
> > Will the file generated always work? What are the current
> > restrictions on kconfig variable names?
> >
> >
> The file generated will always work. It's loaded with makefile's "include"
> directive which is quite straightforward.
> 
> If you're referring to  CONFIG_* variables, I can't find any restriction
> regarding size or anything else.
> If you're referring to the newly created makefile variable foo_KCONF I only
> found this in the documentation:
> "A variable name may be any sequence of characters not containing ‘:’, ‘#’,
> ‘=’, or whitespace."

Both since we'd up with a file with tons of entries like:

bar_KCONF=CONFIG_BAR
foo_KCONF=CONFIG_FOO

I was asking really what checks do we have to ensure either side in these lines
will not make a Makefile barf. For instance using "\" will surely barf. Does
Kconfig check for sanity ? If so then what are the rules for Kconfig variables?
Makefiles then have the above description you mentioned, and I was curious how
and if we want to vet for correctness for both.

> "It is traditional to use upper case letters in variable names, but we
> recommend using lower case letters for variable names that serve internal
> purposes in the makefile, and reserving upper case for parameters that
> control implicit rules or for parameters that the user should override with
> command options"
> 
> Since the module name is lowercase I put the suffix uppercase in order to
> be more visible.

OK thanks.

> > This is nice for experimentation, however as you note we want
> > to evaluate a better way to do this. While at it, can you describe
> > what algorithm currently is used by scripts/kconfig/streamline_config.pl
> > to collect %objects and the associated CONFIG symbol ? Can you think
> > of possible flaws to it ?
> >
> >
> Shortly, all makefiles are parsed searching for patterns such as:
> 
> obj-$(CONFIG_FOO) := obj-foo1.o obj-foo2.o ... etc.
> 
> Each obj-foo is added in the hashmap as key with the value
> array(CONFIG_FOO).
> If obj-foo already exists in the hashmap CONFIG_FOO is appended to the
> array value.

So this fails to then capture lib-$(CONFIG_BAR) then. These can be modules
too. So we'd need something to loop over all possible valid targets to be
able to easily extend this with future targets, in case we get something
other than obj- and lib-. For instance I am looking to add a force-obj-y
and force-lib-y, later this may mean supporting force-obj-m and foce-lib-m,
the actual name however is still up in the air [0], however my point is
we want to be easily able to extend the series of targets used for modules.

This may prove useful to the build something for other things.

Also, does the scripts/kconfig/streamline_config.pl algorithm support
where you have more than one line on a target, for instance:

obj-$(CONFIG_FOO) := foo1.o \
		     foo2.o \
		     foo3.o

If not we'd need to figure out a way to capture these then?

[0] https://lkml.kernel.org/r/20160822235910.GW3296@wotan.suse.de

> I noted that there are situations where an object obj-foo has as associates
> more identical CONFIG_FOOs.
> Since I select only the names which have exaclty one CONFIG_* associated
> this is a bit confusing. I am missing one category of drivers which can be
> found:
> 
> driver-foo ---- CONFIG_FOO CONFIG_FOO

What do these 3 columns represent here ?

> For example: nfit CONFIG_ACPI_NFIT CONFIG_ACPI_NFIT -- this actually
> results in  a module in /sys which does not have a content into
> kconfig_ksymb although it has exactly one match.
> 
> Another thing I can think of is that sometimes we find lines such as:
> 
> foo-name-$(CONFIG_FOO) = obj-foo1.o obj-foo2.o obj-foo3.o
> 
> This can be a clue that the final kernel module is named foo-name and could
> solve some ambiguous cases.
> This are just a few situations I can think of to be kept in mind for the
> next Module.ksymb generator

Indeed thanks, I think it'd be nice if we can iron all of them out. If we cannot
then we should annotate why we can't and see if we are willing to compromise
to help address this.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter
  2016-08-20 15:11     ` Cristina-Gabriela Moraru
@ 2016-08-23 19:07       ` Luis R. Rodriguez
  0 siblings, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-23 19:07 UTC (permalink / raw)
  To: Cristina-Gabriela Moraru
  Cc: Luis R. Rodriguez, linux-kernel, Tom Gundersen, Kay Sievers,
	Rusty Russell, akpm, backports

On Sat, Aug 20, 2016 at 05:11:37PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> > On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> >> Add  CONFIG symbol to kernel modules as a define via -D
> >
> > Perhaps better worded as:
> >
> > When modules have a direct Kconfig CONFIG_ symbol associated with
> > we want to be able to make it available to the build system when we
> > are building the module.
> >
> > You can then describe you do this with -D.
> >
> >> compilation parameter. The CONFIG_FOO symbol for each
> >> module is determined by the module name, using the
> >> associations from Module.ksymb. This file is loaded
> >> using the 'include' directive, thus run like a regular
> >> makefile. The format of the content of the file is the
> >> following:
> >>
> >> foo_KCONF=CONFIG_FOO
> >>
> >> creating a set of Makefile variables foo_KCONF with the
> >> CONFIG_FOO as values.
> >>
> >> The Makefile then adds this value in the compilation
> >> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
> >
> > Very nice. What would it mean if it lacks this ?
> > This should be explained on the commit log.
> >
> 
> If we lack it then KBUILD_KCONF="" and in /sys the module has the
> kconfig_symbol attribute but with the empty string as content:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$
> 
> I will add it into the commit log.

OK -- I do wonder if instead of an empty string leaving the kconfig_symbol
out is better. If its empty then it can be confusing, and so perhaps better
an "unknown" is better. But skipping the attribute then seems best as then
we can focus on just addressing what it *does mean* when we do have a mapping.
This would allow addressing the semantic gap of modules that lack this and
trying to fix those step by step. To fix those we first need to identify
*why* we can't get an attribute pegged to these - document this perhaps on
kernelnewbies.org/KernelProjects/<pick-a-topic-name-for-your-project>
and then your commit log can reference this for a list of known reasons
and pending work.

> >> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> >> index e7df0f5..8ae9b7f 100644
> >> --- a/scripts/Makefile.lib
> >> +++ b/scripts/Makefile.lib
> >> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
> >>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
> >>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
> >>
> >> +# Include Module.ksymb which contains the associations of modules' names
> >> +# and corresponding CONFIG_* options
> >> +include Module.ksymb
> >
> > Is this file always generated? If not prefixing the include call with - would
> > be better. If we are going to make this optional, then an ifdef wrapper would
> > suffice as we know it'd be expected only if the new feature was enabled.
> 
> Yes. This file is always generated. In case a 'git pull' happened
> between two 'make' commands, the associations should be updated. Ok. I
> will add a ifdef.
> 
> >
> >> +
> >>  # These flags are needed for modversions and compiling, so we define them here
> >>  # already
> >>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> >> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> >> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> >> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> >> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> >
> > Are clashes possible with this formula? Can we end up with two results for instance?
> > If not what prevents current konfig logic and namespace from a clash ? If we do
> > not have anything to prevent a clash, what can we do to help make such clash not
> > possible ?
> 
> I think the fact that modname is unique prevents from having clashes.
> KBUILD_KSYMB is found according to modname.
> Currently there is no clash because I enforced the Module.ksymb to
> have 1-to-1 mapping.
> The only potential clash I can imagine right now it having the same
> module name but architecture specific CONFIG_* symbol. If there is a
> 1-to-1 mapping in Module.ksymb there should be no clash in the
> makefile.

OK thanks.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
  2016-08-20 15:16     ` Cristina-Gabriela Moraru
@ 2016-08-23 19:10       ` Luis R. Rodriguez
  0 siblings, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-23 19:10 UTC (permalink / raw)
  To: Cristina-Gabriela Moraru
  Cc: Luis R. Rodriguez, linux-kernel, Tom Gundersen, Kay Sievers,
	Rusty Russell, akpm, Steven Rostedt, backports, vegard.nossum

On Sat, Aug 20, 2016 at 05:16:50PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> >
> > On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > > index 48958d3..a105916 100644
> > > --- a/scripts/mod/modpost.c
> > > +++ b/scripts/mod/modpost.c
> > > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> > >       }
> > >  }
> > >
> > > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > > +{
> > > +     buf_printf(b, "\n");
> > > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
> >
> > What if its not available? What happens?
> >
> 
> If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
> More concrete:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$

As I noted in my other e-mail I think its best we just avoid exposing this
then for modules that lack a mapping, and then we make this best effort,
but document the shortcomings very well and super clearly as to why a
backward map is not possible. This then enables us to do work on the build
system to help with this and complete the semantic gap.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-22 19:35   ` Cristina-Gabriela Moraru
@ 2016-08-23 19:17     ` Luis R. Rodriguez
  0 siblings, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-23 19:17 UTC (permalink / raw)
  To: Cristina-Gabriela Moraru
  Cc: Luis R. Rodriguez, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, Michal Marek, linux-kernel,
	Tom Gundersen, Kay Sievers, Rusty Russell, akpm, backports,
	Guenter Roeck, Greg Kroah-Hartman, rafael.j.wysocki,
	Dmitry Torokhov, Takashi Iwai, Christoph Hellwig,
	Mauro Carvalho Chehab, Johannes Berg, Hauke Mehrtens, Paul Bolle,
	Paul Gortmaker, Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

On Mon, Aug 22, 2016 at 09:35:47PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> >
> >> This patchset implements dynamic pegging of kconfig symbol
> >> into driver modinfo section
> >
> > First a little bit of motivation here helps, so let me try to
> > help fill in some gaps. This may help explain what you have
> > been working on a bit more.
> >
> > First, for those that were not Cc'd but curious about this work
> > so far, you can read the patches here:
> >
> > Original cover letter:
> >
> > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> >
> > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> >
> > There are a few situations in which you may want to extract a
> > subset of Kconfig CONFIG_* symbols for a system. At least when
> > only considering modules:
> >
> >  a) When optimizing build requirements for a kernel for a system.
> >     That is you boot into a distro kernel and then want to build
> >     a slim kernel only with sensible kernel configuration options.
> >
> >  b) When you are on a distribution kernel but the distribution
> >     kernel provided lacks hardware support for your device, you
> >     may either want to upgrade the full kernel in which case you
> >     want to do a) or -- you may want to just a backports release
> >     which provides just the modules you need, you'd use it on top
> >     of the distribution kernel.
> >
> > Are there other *uses* outside of the two above for having a direct
> > deterministic mapping between module and a kconfig symbol ? What
> > if we could also do this for built-in code as well ? kconfig-sat [0] is
> > work that is still being ironed out, but if that was in place and we
> > handed it a set of required options we want, it should in theory help
> > resolve the dependencies to give us a functional .config for a kernel.
> > This of course is a long term possible goal just used as an example
> > -- but are there other uses?
> >
> I can't think of any examples. I will add this ones in the cover
> letter as motivation.

Michal provided one more :) -- which was actually a side goal you had I think.

Very clearly documenting the motivation is very important here, side benefits
that you can think of are (some listed above) should help understand why
we may went to or not do some further enhancements to build system to help
with more deterministic backwards mappings.

> > To be clear, you only peg the CONFIG_ option if you are
> > 100% certain of having a direct mapping, is that right ?
> >
> 
> Yes.

OK but as you clarified in other e-mail then the attribute is added but
its just empty. This is rather ambiguous -- lets instead try to not add
the attribute if one is not available.

> And there are some drivers which have duplicate values in the hash.
> foo.o -> CONFIG_FOO CONFIG_FOO

What is the reason for this ?

> Although this one has one match it's filtered as having 2 CONFIGs.
> 
> >> The result of this patchset is the following. All modules from
> >> /sys expose the correct CONFIG_* symbol in the module attribute
> >> kconfig_symb with some exceptions. For a total number of 58
> >> modules,
> >
> > What kernel configuration did you use ? What architecture ?
> > If you use allmodconfig, what do the numbers look like ?
> >
> 
> I have a virtual machine in which I created the configuration with
> make localmodconfig and installed the new kernel.
> $  uname -a
> Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
> x86_64 x86_64 GNU/Linux
> 
> Should I send you the .config file ?

This is not needed, I ran your patches through 0-day and it was happy.
Good job !

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-19  9:07   ` Michal Marek
  2016-08-22 19:48     ` Cristina-Gabriela Moraru
@ 2016-08-23 21:32     ` Luis R. Rodriguez
  2016-08-24 11:05       ` Michal Marek
  1 sibling, 1 reply; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-23 21:32 UTC (permalink / raw)
  To: Michal Marek
  Cc: Luis R. Rodriguez, Cristina Moraru, vegard.nossum,
	Valentin Rothberg, Hannes Reinecke, Sam Ravnborg, linux-kernel,
	teg, kay, rusty, akpm, backports, Guenter Roeck,
	Greg Kroah-Hartman, rafael.j.wysocki, Dmitry Torokhov,
	Takashi Iwai, Christoph Hellwig, Mauro Carvalho Chehab,
	Johannes Berg, Hauke Mehrtens, Paul Bolle, Paul Gortmaker,
	Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> > 
> >> This patchset implements dynamic pegging of kconfig symbol
> >> into driver modinfo section
> > 
> > First a little bit of motivation here helps, so let me try to
> > help fill in some gaps. This may help explain what you have
> > been working on a bit more.
> > 
> > First, for those that were not Cc'd but curious about this work
> > so far, you can read the patches here:
> > 
> > Original cover letter:
> > 
> > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> > 
> > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> > 
> > There are a few situations in which you may want to extract a
> > subset of Kconfig CONFIG_* symbols for a system. At least when
> > only considering modules:
> > 
> >  a) When optimizing build requirements for a kernel for a system.
> >     That is you boot into a distro kernel and then want to build
> >     a slim kernel only with sensible kernel configuration options.
> > 
> >  b) When you are on a distribution kernel but the distribution
> >     kernel provided lacks hardware support for your device, you
> >     may either want to upgrade the full kernel in which case you
> >     want to do a) or -- you may want to just a backports release
> >     which provides just the modules you need, you'd use it on top
> >     of the distribution kernel.
> 
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl.

Indeed, this was actually a side possible goal here :) only it seems
we totally forgot about it while addressing the work.

> Only if the patch did not depend on streamline_config.pl :)

Indeed, lets get rid of that dependency.

> One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
> 
> CONFIG_FOO=m-CONFIG_FOO
> 
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.

Neat, would that cover lib-m then as well or other targets that would generate 
modules ?

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.

Well so one future use would be for instance a 'make udevconfig' which
in theory should be able then to scrape all possible related needed
device CONFIG_'s and then give you an optimal kernel configuration.
It'd be nice to be able to extract this from the kernel and modules
without needing build sources in /lib/modules/`uname -r`/.

Thoughts?

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-23 21:32     ` Luis R. Rodriguez
@ 2016-08-24 11:05       ` Michal Marek
  2016-08-24 16:33         ` Luis R. Rodriguez
  0 siblings, 1 reply; 28+ messages in thread
From: Michal Marek @ 2016-08-24 11:05 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Cristina Moraru, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, linux-kernel, teg, kay, rusty,
	akpm, backports, Guenter Roeck, Greg Kroah-Hartman,
	rafael.j.wysocki, Dmitry Torokhov, Takashi Iwai,
	Christoph Hellwig, Mauro Carvalho Chehab, Johannes Berg,
	Hauke Mehrtens, Paul Bolle, Paul Gortmaker, Alexey Khoroshilov,
	Sathya Prakash Veerichetty, Martin K. Petersen, Laurence Oberman,
	Johannes Thumshirn, Tejun Heo, Jej B, Theodore Ts'o,
	danijons, Andrzej Wasowski

On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>>
>>>> This patchset implements dynamic pegging of kconfig symbol
>>>> into driver modinfo section
>>>
>>> First a little bit of motivation here helps, so let me try to
>>> help fill in some gaps. This may help explain what you have
>>> been working on a bit more.
>>>
>>> First, for those that were not Cc'd but curious about this work
>>> so far, you can read the patches here:
>>>
>>> Original cover letter:
>>>
>>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>>
>>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>>
>>> There are a few situations in which you may want to extract a
>>> subset of Kconfig CONFIG_* symbols for a system. At least when
>>> only considering modules:
>>>
>>>  a) When optimizing build requirements for a kernel for a system.
>>>     That is you boot into a distro kernel and then want to build
>>>     a slim kernel only with sensible kernel configuration options.
>>>
>>>  b) When you are on a distribution kernel but the distribution
>>>     kernel provided lacks hardware support for your device, you
>>>     may either want to upgrade the full kernel in which case you
>>>     want to do a) or -- you may want to just a backports release
>>>     which provides just the modules you need, you'd use it on top
>>>     of the distribution kernel.
>>
>> c) Having the mapping in sysfs would allow to simplify
>> streamline_config.pl avoid parsing Makefiles in perl.
> 
> Indeed, this was actually a side possible goal here :) only it seems
> we totally forgot about it while addressing the work.
> 
>> Only if the patch did not depend on streamline_config.pl :)
> 
> Indeed, lets get rid of that dependency.
> 
>> One idea would be to generate
>> the Module.ksymb in a similar way we generate the modules.builtin file:
>> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> replaced with
>>
>> CONFIG_FOO=m-CONFIG_FOO
>>
>> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
                                       should be obj-m-CONFIG_%

>> create the mapping. This would also properly cover cases where we build
>> the $(obj-m) list from another list. It would certainly create other
>> corner cases, but it's worth trying IMO.
> 
> Neat, would that cover lib-m then as well or other targets that would generate 
> modules ?

This would have to be handled as well by the patch. Obviously it's not
going to be as simple as described in a single paragraph of text :).


>> Another thing is that we do not necessarily need to record this
>> information in .modinfo, but we can generate a list in
>> /lib/modules/`uname -r`/ for consumption. It could also include drivers
>> that are builtin in the current configuration.
> 
> Well so one future use would be for instance a 'make udevconfig' which
> in theory should be able then to scrape all possible related needed
> device CONFIG_'s and then give you an optimal kernel configuration.
> It'd be nice to be able to extract this from the kernel and modules
> without needing build sources in /lib/modules/`uname -r`/.

I meant that the file can be installed by modules_install. Like you have
/lib/modules/`uname -r`/modules.builtin with recent kernels and recent
distro packages. But really, it's a minor issue how the information is
presented to userspace.

Michal
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-24 11:05       ` Michal Marek
@ 2016-08-24 16:33         ` Luis R. Rodriguez
  2016-08-24 17:31           ` Naveen Kumar
  0 siblings, 1 reply; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-24 16:33 UTC (permalink / raw)
  To: Michal Marek
  Cc: Luis R. Rodriguez, Cristina Moraru, vegard.nossum,
	Valentin Rothberg, Hannes Reinecke, Sam Ravnborg, linux-kernel,
	teg, kay, rusty, akpm, backports, Guenter Roeck,
	Greg Kroah-Hartman, rafael.j.wysocki, Dmitry Torokhov,
	Takashi Iwai, Christoph Hellwig, Mauro Carvalho Chehab,
	Johannes Berg, Hauke Mehrtens, Paul Bolle, Paul Gortmaker,
	Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> >>>
> >>>> This patchset implements dynamic pegging of kconfig symbol
> >>>> into driver modinfo section
> >>>
> >>> First a little bit of motivation here helps, so let me try to
> >>> help fill in some gaps. This may help explain what you have
> >>> been working on a bit more.
> >>>
> >>> First, for those that were not Cc'd but curious about this work
> >>> so far, you can read the patches here:
> >>>
> >>> Original cover letter:
> >>>
> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> >>>
> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> >>>
> >>> There are a few situations in which you may want to extract a
> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
> >>> only considering modules:
> >>>
> >>>  a) When optimizing build requirements for a kernel for a system.
> >>>     That is you boot into a distro kernel and then want to build
> >>>     a slim kernel only with sensible kernel configuration options.
> >>>
> >>>  b) When you are on a distribution kernel but the distribution
> >>>     kernel provided lacks hardware support for your device, you
> >>>     may either want to upgrade the full kernel in which case you
> >>>     want to do a) or -- you may want to just a backports release
> >>>     which provides just the modules you need, you'd use it on top
> >>>     of the distribution kernel.
> >>
> >> c) Having the mapping in sysfs would allow to simplify
> >> streamline_config.pl avoid parsing Makefiles in perl.
> > 
> > Indeed, this was actually a side possible goal here :) only it seems
> > we totally forgot about it while addressing the work.
> > 
> >> Only if the patch did not depend on streamline_config.pl :)
> > 
> > Indeed, lets get rid of that dependency.
> > 
> >> One idea would be to generate
> >> the Module.ksymb in a similar way we generate the modules.builtin file:
> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> >> replaced with
> >>
> >> CONFIG_FOO=m-CONFIG_FOO
> >>
> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
>                                        should be obj-m-CONFIG_%
> 
> >> create the mapping. This would also properly cover cases where we build
> >> the $(obj-m) list from another list. It would certainly create other
> >> corner cases, but it's worth trying IMO.
> > 
> > Neat, would that cover lib-m then as well or other targets that would generate 
> > modules ?
> 
> This would have to be handled as well by the patch. Obviously it's not
> going to be as simple as described in a single paragraph of text :).

Ah OK, it was not clear to me if your suggestion captured all modules already.
There may still be a way. We just have to look harder. Then what precise
target was used would be irrelevant.

> >> Another thing is that we do not necessarily need to record this
> >> information in .modinfo, but we can generate a list in
> >> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> >> that are builtin in the current configuration.
> > 
> > Well so one future use would be for instance a 'make udevconfig' which
> > in theory should be able then to scrape all possible related needed
> > device CONFIG_'s and then give you an optimal kernel configuration.
> > It'd be nice to be able to extract this from the kernel and modules
> > without needing build sources in /lib/modules/`uname -r`/.
> 
> I meant that the file can be installed by modules_install. Like you have
> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
> distro packages.

Got it.

> But really, it's a minor issue how the information is
> presented to userspace.

To us perhaps, but to userspace and users this is a big deal. It'd be nice
to be able to construct a kernel config only with what CONFIG's your hardware
needs, doing this should mean trying to expose enough information in a way
userspace is easily able to handle. The reason to go with the mod section stuff
was this was already being picked up by udevadm for its own hardware db info,
so this seemed like a logical evolution, and it would not require your
kernel sources installed.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-24 16:33         ` Luis R. Rodriguez
@ 2016-08-24 17:31           ` Naveen Kumar
  0 siblings, 0 replies; 28+ messages in thread
From: Naveen Kumar @ 2016-08-24 17:31 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Michal Marek, Cristina Moraru, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, linux-kernel, teg, kay, rusty,
	akpm, backports, Guenter Roeck, Greg Kroah-Hartman,
	rafael.j.wysocki, Dmitry Torokhov, Takashi Iwai,
	Christoph Hellwig, Mauro Carvalho Chehab, Johannes Berg,
	Hauke Mehrtens, Paul Bolle, Paul Gortmaker, Alexey Khoroshilov,
	Sathya Prakash Veerichetty, Martin K. Petersen, Laurence Oberman,
	Johannes Thumshirn, Tejun Heo, Jej B, Theodore Ts'o,
	danijons, Andrzej Wasowski

unsubscribe backports

On 8/24/16, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
>> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
>> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>> >>>
>> >>>> This patchset implements dynamic pegging of kconfig symbol
>> >>>> into driver modinfo section
>> >>>
>> >>> First a little bit of motivation here helps, so let me try to
>> >>> help fill in some gaps. This may help explain what you have
>> >>> been working on a bit more.
>> >>>
>> >>> First, for those that were not Cc'd but curious about this work
>> >>> so far, you can read the patches here:
>> >>>
>> >>> Original cover letter:
>> >>>
>> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>> >>>
>> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>> >>>
>> >>> There are a few situations in which you may want to extract a
>> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> >>> only considering modules:
>> >>>
>> >>>  a) When optimizing build requirements for a kernel for a system.
>> >>>     That is you boot into a distro kernel and then want to build
>> >>>     a slim kernel only with sensible kernel configuration options.
>> >>>
>> >>>  b) When you are on a distribution kernel but the distribution
>> >>>     kernel provided lacks hardware support for your device, you
>> >>>     may either want to upgrade the full kernel in which case you
>> >>>     want to do a) or -- you may want to just a backports release
>> >>>     which provides just the modules you need, you'd use it on top
>> >>>     of the distribution kernel.
>> >>
>> >> c) Having the mapping in sysfs would allow to simplify
>> >> streamline_config.pl avoid parsing Makefiles in perl.
>> >
>> > Indeed, this was actually a side possible goal here :) only it seems
>> > we totally forgot about it while addressing the work.
>> >
>> >> Only if the patch did not depend on streamline_config.pl :)
>> >
>> > Indeed, lets get rid of that dependency.
>> >
>> >> One idea would be to generate
>> >> the Module.ksymb in a similar way we generate the modules.builtin
>> >> file:
>> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> >> replaced with
>> >>
>> >> CONFIG_FOO=m-CONFIG_FOO
>> >>
>> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES))
>> >> to
>>                                        should be obj-m-CONFIG_%
>>
>> >> create the mapping. This would also properly cover cases where we
>> >> build
>> >> the $(obj-m) list from another list. It would certainly create other
>> >> corner cases, but it's worth trying IMO.
>> >
>> > Neat, would that cover lib-m then as well or other targets that would
>> > generate
>> > modules ?
>>
>> This would have to be handled as well by the patch. Obviously it's not
>> going to be as simple as described in a single paragraph of text :).
>
> Ah OK, it was not clear to me if your suggestion captured all modules
> already.
> There may still be a way. We just have to look harder. Then what precise
> target was used would be irrelevant.
>
>> >> Another thing is that we do not necessarily need to record this
>> >> information in .modinfo, but we can generate a list in
>> >> /lib/modules/`uname -r`/ for consumption. It could also include
>> >> drivers
>> >> that are builtin in the current configuration.
>> >
>> > Well so one future use would be for instance a 'make udevconfig' which
>> > in theory should be able then to scrape all possible related needed
>> > device CONFIG_'s and then give you an optimal kernel configuration.
>> > It'd be nice to be able to extract this from the kernel and modules
>> > without needing build sources in /lib/modules/`uname -r`/.
>>
>> I meant that the file can be installed by modules_install. Like you have
>> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
>> distro packages.
>
> Got it.
>
>> But really, it's a minor issue how the information is
>> presented to userspace.
>
> To us perhaps, but to userspace and users this is a big deal. It'd be nice
> to be able to construct a kernel config only with what CONFIG's your
> hardware
> needs, doing this should mean trying to expose enough information in a way
> userspace is easily able to handle. The reason to go with the mod section
> stuff
> was this was already being picked up by udevadm for its own hardware db
> info,
> so this seemed like a logical evolution, and it would not require your
> kernel sources installed.
>
>   Luis
> --
> To unsubscribe from this list: send the line "unsubscribe backports" in
>
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-18 17:55 ` [RFC PATCH 0/5] Add CONFIG symbol as module attribute Luis R. Rodriguez
  2016-08-19  9:07   ` Michal Marek
  2016-08-22 19:35   ` Cristina-Gabriela Moraru
@ 2016-08-25  7:43   ` Christoph Hellwig
  2016-08-25  8:00     ` Johannes Berg
                       ` (2 more replies)
  2 siblings, 3 replies; 28+ messages in thread
From: Christoph Hellwig @ 2016-08-25  7:43 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Cristina Moraru, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, Michal Marek, linux-kernel, teg,
	kay, rusty, akpm, backports, Guenter Roeck, Greg Kroah-Hartman,
	rafael.j.wysocki, Dmitry Torokhov, Takashi Iwai,
	Christoph Hellwig, Mauro Carvalho Chehab, Johannes Berg,
	Hauke Mehrtens, Paul Bolle, Paul Gortmaker, Alexey Khoroshilov,
	Sathya Prakash Veerichetty, Martin K. Petersen, Laurence Oberman,
	Johannes Thumshirn, Tejun Heo, Jej B, Theodore Ts'o,
	danijons, Andrzej Wasowski

The idea seems useful, but I reallt don't like the 'reverse-engineering'
approach.

If we want to this properly from the ground up we should just split out
our CONFIG_ SYMBOLS into

MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
only one, then like a bool)

CONFIG_* - just bool, MODULE_ may depend on it, too.

The other nice thing is that we could probably fold most of the Makefiles
into Kconfig using that methods as well, by listing the objectes required
for a module, e.g.

module NVME_TARGET
	tristate "NVMe Target support"
	depends on BLOCK
	depends on CONFIGFS_FS
	name nvmet
	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
	objects discovery.o

module NVME_TARGET_LOOP
	tristate "NVMe loopback device support"
	depends on BLK_DEV_NVME
	depends on NVME_TARGET
	select NVME_FABRICS
	select SG_POOL
	name nvme-loop
	objects loop.o
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-25  7:43   ` Christoph Hellwig
@ 2016-08-25  8:00     ` Johannes Berg
  2016-08-25 19:51       ` Luis R. Rodriguez
  2016-08-25  8:41     ` Michal Marek
  2016-08-25 20:19     ` Luis R. Rodriguez
  2 siblings, 1 reply; 28+ messages in thread
From: Johannes Berg @ 2016-08-25  8:00 UTC (permalink / raw)
  To: Christoph Hellwig, Luis R. Rodriguez
  Cc: Cristina Moraru, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, Michal Marek, linux-kernel, teg,
	kay, rusty, akpm, backports, Guenter Roeck, Greg Kroah-Hartman,
	rafael.j.wysocki, Dmitry Torokhov, Takashi Iwai,
	Mauro Carvalho Chehab, Hauke Mehrtens, Paul Bolle,
	Paul Gortmaker, Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski


> The other nice thing is that we could probably fold most of the
> Makefiles into Kconfig using that methods as well, by listing the
> objectes required for a module, e.g.
> 
> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 

If this was going to be a thing, then you might also have

config NVME_TARGET_FOO
	bool "NVMe target supports FOO"
	module NVME_TARGET
	objects foo.o

The "module" would be like a "depends on" plus giving the module for
generating the Makefile, and now you can really remove most Makefile
stuff... :)

johannes
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-25  7:43   ` Christoph Hellwig
  2016-08-25  8:00     ` Johannes Berg
@ 2016-08-25  8:41     ` Michal Marek
  2016-08-25 20:19     ` Luis R. Rodriguez
  2 siblings, 0 replies; 28+ messages in thread
From: Michal Marek @ 2016-08-25  8:41 UTC (permalink / raw)
  To: Christoph Hellwig, Luis R. Rodriguez
  Cc: Cristina Moraru, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, linux-kernel, teg, kay, rusty,
	akpm, backports, Guenter Roeck, Greg Kroah-Hartman,
	rafael.j.wysocki, Dmitry Torokhov, Takashi Iwai,
	Mauro Carvalho Chehab, Johannes Berg, Hauke Mehrtens, Paul Bolle,
	Paul Gortmaker, Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

On 2016-08-25 09:43, Christoph Hellwig wrote:
> The idea seems useful, but I reallt don't like the 'reverse-engineering'
> approach.
> 
> If we want to this properly from the ground up we should just split out
> our CONFIG_ SYMBOLS into
> 
> MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> only one, then like a bool)
> 
> CONFIG_* - just bool, MODULE_ may depend on it, too.
> 
> The other nice thing is that we could probably fold most of the Makefiles
> into Kconfig using that methods as well, by listing the objectes required
> for a module, e.g.
> 
> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 
> module NVME_TARGET_LOOP
> 	tristate "NVMe loopback device support"
> 	depends on BLK_DEV_NVME
> 	depends on NVME_TARGET
> 	select NVME_FABRICS
> 	select SG_POOL
> 	name nvme-loop
> 	objects loop.o

It looks nice for the simple case, but as soon as you need something
non-standard, you will need to mix a kconfig generated and a manually
written Makefile snippet.

Michal
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-25  8:00     ` Johannes Berg
@ 2016-08-25 19:51       ` Luis R. Rodriguez
  0 siblings, 0 replies; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-25 19:51 UTC (permalink / raw)
  To: Johannes Berg
  Cc: Christoph Hellwig, Luis R. Rodriguez, Cristina Moraru,
	vegard.nossum, Valentin Rothberg, Hannes Reinecke, Sam Ravnborg,
	Michal Marek, linux-kernel, teg, kay, rusty, akpm, backports,
	Guenter Roeck, Greg Kroah-Hartman, rafael.j.wysocki,
	Dmitry Torokhov, Takashi Iwai, Mauro Carvalho Chehab,
	Hauke Mehrtens, Paul Bolle, Paul Gortmaker, Alexey Khoroshilov,
	Sathya Prakash Veerichetty, Martin K. Petersen, Laurence Oberman,
	Johannes Thumshirn, Tejun Heo, Jej B, Theodore Ts'o,
	danijons, Andrzej Wasowski

On Thu, Aug 25, 2016 at 10:00:20AM +0200, Johannes Berg wrote:
> 
> > The other nice thing is that we could probably fold most of the
> > Makefiles into Kconfig using that methods as well, by listing the
> > objectes required for a module, e.g.
> > 
> > module NVME_TARGET
> > 	tristate "NVMe Target support"
> > 	depends on BLOCK
> > 	depends on CONFIGFS_FS
> > 	name nvmet
> > 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> > 	objects discovery.o
> > 
> 
> If this was going to be a thing, then you might also have
> 
> config NVME_TARGET_FOO
> 	bool "NVMe target supports FOO"
> 	module NVME_TARGET
> 	objects foo.o

You mean this instead of a Makefile:

foo-$(CONFIG_NVME_TARGET_FOO) += foo.o

?

> The "module" would be like a "depends on" plus giving the module for
> generating the Makefile, and now you can really remove most Makefile
> stuff... :)

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-25  7:43   ` Christoph Hellwig
  2016-08-25  8:00     ` Johannes Berg
  2016-08-25  8:41     ` Michal Marek
@ 2016-08-25 20:19     ` Luis R. Rodriguez
  2019-02-05 22:07       ` Luis Chamberlain
  2 siblings, 1 reply; 28+ messages in thread
From: Luis R. Rodriguez @ 2016-08-25 20:19 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Luis R. Rodriguez, Cristina Moraru, vegard.nossum,
	Valentin Rothberg, Hannes Reinecke, Sam Ravnborg, Michal Marek,
	linux-kernel, teg, kay, rusty, akpm, backports, Guenter Roeck,
	Greg Kroah-Hartman, rafael.j.wysocki, Dmitry Torokhov,
	Takashi Iwai, Mauro Carvalho Chehab, Johannes Berg,
	Hauke Mehrtens, Paul Bolle, Paul Gortmaker, Alexey Khoroshilov,
	Sathya Prakash Veerichetty, Martin K. Petersen, Laurence Oberman,
	Johannes Thumshirn, Tejun Heo, Jej B, Theodore Ts'o,
	danijons, Andrzej Wasowski

On Thu, Aug 25, 2016 at 09:43:13AM +0200, Christoph Hellwig wrote:
> The idea seems useful, but I reallt don't like the 'reverse-engineering'
> approach.
> 
> If we want to this properly from the ground up we should just split out
> our CONFIG_ SYMBOLS into
> 
> MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> only one, then like a bool)
> 
> CONFIG_* - just bool, MODULE_ may depend on it, too.

Curious what does the split buy us if the real meaningful input is the value
assigned to the config ? Ie, MODULE_FOO=m would be the modules we want to
check for.

> The other nice thing is that we could probably fold most of the Makefiles
> into Kconfig using that methods as well, by listing the objectes required
> for a module, e.g.

OK If the Kconfig file has the objects listed I can see the gain of using
Kconfig then to more easily map out to a symbol, given doing this on Makefiles
is not straight forward.

> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 
> module NVME_TARGET_LOOP
> 	tristate "NVMe loopback device support"
> 	depends on BLK_DEV_NVME
> 	depends on NVME_TARGET
> 	select NVME_FABRICS
> 	select SG_POOL
> 	name nvme-loop
> 	objects loop.o

I can see a huge win of having a direct specification that provides as a
feature two way mapping from CONFIG <--> module (objects) and backwards again
easily and clearly without hacks, specially if upon boot then we can then
provide the precise kernel configuration you need, for both built-in and
modules. The above could help with modules -- for built-in reverse mapping
we'd need something else, perhaps a configurable option to keep tabs on
inits called with associated configs.

The above would be a pretty intrusive change though, in comparison to
Cristina's original approach. The reverse-engineering object --> config
aspect of her work and of the old scripts/kconfig/streamline_config.pl
explains why it was hard. I'd be curious to learn of other gains possible
other than those listed so far, if we had this.

Re-iterating gains of having a simple two way CONFIG <--> module (objects)
mapping (following the above proposal now):

a) When optimizing build requirements for a kernel for a system.                                                                                                           
   That is you boot into a distro kernel and then want to build                                                                                                            
   a slim kernel only with sensible kernel configuration options.                                                                                                          

b) When you are on a distribution kernel but the distribution                                                                                                              
   kernel provided lacks hardware support for your device, you                                                                                                             
   may either want to upgrade the full kernel in which case you                                                                                                            
   want to do a) or -- you may want to just a backports release                                                                                                            
   which provides just the modules you need, you'd use it on top                                                                                                           
   of the distribution kernel.                                                                                                                                             

c) Having the mapping in sysfs would allow to simplify                                                                                                                       
   streamline_config.pl avoid parsing Makefiles in perl. (From Michal)

d) Fold most of the Makefiles into Kconfig

In retrospect c) still seems related to a) as we'd do away with
the hacks completely needed by streamline_config.pl, a) can be
augmented if we figure out a built-in solution as well.

d) Just seems like collateral of a more precise mapping than
what a Makefile provides. Anything else ?

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute
  2016-08-25 20:19     ` Luis R. Rodriguez
@ 2019-02-05 22:07       ` Luis Chamberlain
  0 siblings, 0 replies; 28+ messages in thread
From: Luis Chamberlain @ 2019-02-05 22:07 UTC (permalink / raw)
  To: Christoph Hellwig, Brendan Higgins
  Cc: Cristina Moraru, vegard.nossum, Valentin Rothberg,
	Hannes Reinecke, Sam Ravnborg, Michal Marek, linux-kernel,
	Tom Gundersen, Kay Sievers, Rusty Russell, Andrew Morton,
	backports, Guenter Roeck, Greg Kroah-Hartman, rafael.j.wysocki,
	Dmitry Torokhov, Takashi Iwai, Mauro Carvalho Chehab,
	Johannes Berg, Hauke Mehrtens, Paul Bolle, Paul Gortmaker,
	Alexey Khoroshilov, Sathya Prakash Veerichetty,
	Martin K. Petersen, Laurence Oberman, Johannes Thumshirn,
	Tejun Heo, Jej B, Theodore Ts'o, danijons, Andrzej Wasowski

On Thu, Aug 25, 2016 at 2:19 PM Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>
> On Thu, Aug 25, 2016 at 09:43:13AM +0200, Christoph Hellwig wrote:
> > The idea seems useful, but I reallt don't like the 'reverse-engineering'
> > approach.
> >
> > If we want to this properly from the ground up we should just split out
> > our CONFIG_ SYMBOLS into
> >
> > MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> > only one, then like a bool)
> >
> > CONFIG_* - just bool, MODULE_ may depend on it, too.
>
> Curious what does the split buy us if the real meaningful input is the value
> assigned to the config ? Ie, MODULE_FOO=m would be the modules we want to
> check for.

Also, unless you have kernel sources I don't think you could still
easily infer which MODULE_* config option enabled the module, could
you? And this was one of the original goals with this patch set. I
still think extending the modinfo section would be needed to allow
udevadm info to easily extract the information, no?

> > The other nice thing is that we could probably fold most of the Makefiles
> > into Kconfig using that methods as well, by listing the objectes required
> > for a module, e.g.
>
> OK If the Kconfig file has the objects listed I can see the gain of using
> Kconfig then to more easily map out to a symbol, given doing this on Makefiles
> is not straight forward.
>
> > module NVME_TARGET
> >       tristate "NVMe Target support"
> >       depends on BLOCK
> >       depends on CONFIGFS_FS
> >       name nvmet
> >       objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> >       objects discovery.o
> >
> > module NVME_TARGET_LOOP
> >       tristate "NVMe loopback device support"
> >       depends on BLK_DEV_NVME
> >       depends on NVME_TARGET
> >       select NVME_FABRICS
> >       select SG_POOL
> >       name nvme-loop
> >       objects loop.o
>
> I can see a huge win of having a direct specification that provides as a
> feature two way mapping from CONFIG <--> module (objects) and backwards again
> easily and clearly without hacks, specially if upon boot then we can then
> provide the precise kernel configuration you need, for both built-in and
> modules. The above could help with modules -- for built-in reverse mapping
> we'd need something else, perhaps a configurable option to keep tabs on
> inits called with associated configs.
>
> The above would be a pretty intrusive change though, in comparison to
> Cristina's original approach.

I think this is the biggest issue so far, and that we have no code to
showcase for it. Meanwhile pegging at least a kconfig symbol to
modinfo for at least a lot of modules would help considerably for
other use cases mentioned.

> The reverse-engineering object --> config
> aspect of her work and of the old scripts/kconfig/streamline_config.pl
> explains why it was hard. I'd be curious to learn of other gains possible
> other than those listed so far, if we had this.
>
> Re-iterating gains of having a simple two way CONFIG <--> module (objects)
> mapping (following the above proposal now):
>
> a) When optimizing build requirements for a kernel for a system.
>    That is you boot into a distro kernel and then want to build
>    a slim kernel only with sensible kernel configuration options.
>
> b) When you are on a distribution kernel but the distribution
>    kernel provided lacks hardware support for your device, you
>    may either want to upgrade the full kernel in which case you
>    want to do a) or -- you may want to just a backports release
>    which provides just the modules you need, you'd use it on top
>    of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
>    streamline_config.pl avoid parsing Makefiles in perl. (From Michal)

It took me a while to recall where this was and why after 3 years of
not following up on this thread. So in case others fall into the same
boat, the sysfs gain here is that the modinfo section could be
readable then on any module's:

/sys/module/e1000/section/kconfig_symb

> d) Fold most of the Makefiles into Kconfig
>
> In retrospect c) still seems related to a) as we'd do away with
> the hacks completely needed by streamline_config.pl, a) can be
> augmented if we figure out a built-in solution as well.
>
> d) Just seems like collateral of a more precise mapping than
> what a Makefile provides. Anything else ?

In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
on this, I'm inclined to believe *at least* having some kconfig_symb
exposed for some modules is better than nothing. Christoph are you
totally opposed to this effort until we get a non-reverse engineered
effort in place? It just seems like an extraordinary amount of work
and I'm not quite sure who's volunteering to do it.

Other stakeholders may benefit from at least having some config -->
module mapping for now. Not just backports or building slimmer
kernels.

 Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

^ permalink raw reply	[flat|nested] 28+ messages in thread

end of thread, back to index

Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 17:55 ` [RFC PATCH 0/5] Add CONFIG symbol as module attribute Luis R. Rodriguez
2016-08-19  9:07   ` Michal Marek
2016-08-22 19:48     ` Cristina-Gabriela Moraru
2016-08-23 21:32     ` Luis R. Rodriguez
2016-08-24 11:05       ` Michal Marek
2016-08-24 16:33         ` Luis R. Rodriguez
2016-08-24 17:31           ` Naveen Kumar
2016-08-22 19:35   ` Cristina-Gabriela Moraru
2016-08-23 19:17     ` Luis R. Rodriguez
2016-08-25  7:43   ` Christoph Hellwig
2016-08-25  8:00     ` Johannes Berg
2016-08-25 19:51       ` Luis R. Rodriguez
2016-08-25  8:41     ` Michal Marek
2016-08-25 20:19     ` Luis R. Rodriguez
2019-02-05 22:07       ` Luis Chamberlain
     [not found] ` <1471462023-119645-3-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:10   ` [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter Luis R. Rodriguez
2016-08-18 18:55     ` Luis R. Rodriguez
2016-08-20 15:11     ` Cristina-Gabriela Moraru
2016-08-23 19:07       ` Luis R. Rodriguez
     [not found] ` <1471462023-119645-2-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:22   ` [RFC PATCH 1/5] Add generation of Module.symb in streamline_config Luis R. Rodriguez
2016-08-18 18:32     ` Luis R. Rodriguez
     [not found]     ` <CAGZ2q2xi9Uy-ye387=mWhy_fOEJBC593Nos7fH027m-_ZdoOXA@mail.gmail.com>
2016-08-20 14:49       ` Cristina-Gabriela Moraru
2016-08-23 19:00       ` Luis R. Rodriguez
     [not found] ` <1471462023-119645-4-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:30   ` [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile Luis R. Rodriguez
     [not found] ` <1471462023-119645-5-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 18:59   ` [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute Luis R. Rodriguez
2016-08-20 15:16     ` Cristina-Gabriela Moraru
2016-08-23 19:10       ` Luis R. Rodriguez
     [not found] ` <1471462023-119645-6-git-send-email-cristina.moraru09@gmail.com>
2016-08-18 19:02   ` [RFC PATCH 5/5] Add kconf_symb as kernel " Luis R. Rodriguez

Backports Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/backports/0 backports/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 backports backports/ https://lore.kernel.org/backports \
		backports@vger.kernel.org backports@archiver.kernel.org
	public-inbox-index backports


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.backports


AGPL code for this site: git clone https://public-inbox.org/ public-inbox