[RFC PATCH 0/5] Add CONFIG symbol as module attribute

[RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Cristina Moraru]

This patchset implements dynamic pegging of kconfig symbol
into driver modinfo section

* updates streamline_config.pl to generate the auxiliary file
scripts/mod/Module.ksymb containing associations of driver file
names and corresponding kconfig symbols CONFIG_*
* updates Makefiles to trigger streamline_config.pl call for
Module.ksymb generation and pass the determined CONFIG_* symbol
as compilation parameter -D in KBUILD_KSYMB variable
* adds kconfig_symb as module attribute
* updates modpost to set KBUILD_KSYMB macro as value for
kbuild_symb attribute

Note: the content of the file Module.ksymb is generated for
all modules that have only one associate CONFIG option. All
others are considered to be components linked at the final
modules but not final modules itselves.

The result of this patchset is the following. All modules from
/sys expose the correct CONFIG_* symbol in the module attribute
kconfig_symb with some exceptions. For a total number of 58
modules, 4 of them do not:

snd_seq_midi_event
mptscsih
libahci
mptbase

After a short research:

For mptscsih - ./drivers/message/fusion/Makefile
obj-$(CONFIG_FUSION_SPI)	+= mptbase.o mptscsih.o mptspi.o
obj-$(CONFIG_FUSION_FC)		+= mptbase.o mptscsih.o mptfc.o
obj-$(CONFIG_FUSION_SAS)	+= mptbase.o mptscsih.o mptsas.o
As appears in the Makefile mptscsi is part of more config
options so it's excluded because it's not considered a module
by itself but, somehow gets compiled as module. Same for the
mptbase. I still have to understand better the Kbuild logic so
I cannot state that this is loose practice.

For libahci there are multiple CONFIG_ symbols for different
architectures.

Although past discussion recommended not to use streamline_config,
this patchset still relies on it, for the sake of the proof of
concept. Although the result is close to the target it still does
not provide complete correctness. However it can be replaced by
creating another script which tries to determine better the
module <-> CONFIG associations and output them in auxiliary file
Module.ksymb. Maybe this way we could also determine all CONFIGs
for a particular driver, not only the exact one that enables it.

The auxiliary file is necessary because the Makefile itself does
not have a mapping. The makefile includes the config file with
directive include which creates a series of internal variables
CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
other Makefiles, lines like 

obj-$(CONFIG_FOO) = foo.o 

are resolved in place to obj-y, obj-m etc according to 'make'
logic and the association is lost.

Finally, what does this patchset provide is an infrastructure
to dinamically peg CONFIG_* options to associate drivers using
the mapping from Module.ksymb file. Generation of Module.ksymb
can be replaced but keeping the same format permit the usage of
the other patches.

This patchset is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Cristina Moraru (5):
  Add generation of Module.symb in streamline_config
  Add CONFIG symbol to module as compilation parameter
  Trigger Module.ksymb generation in Makefile
  Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
  Add kconf_symb as kernel module attribute

 Makefile                             |  4 ++++
 include/linux/module.h               |  1 +
 kernel/module.c                      |  2 ++
 scripts/Makefile.lib                 |  9 ++++++++-
 scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
 scripts/mod/modpost.c                |  7 +++++++
 6 files changed, 51 insertions(+), 2 deletions(-)

-- 
2.7.4

[RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Cristina Moraru]

Add generation of ./scripts/Module.ksymb file containing
associations of driver file names and corresponding CONFIG_*
symbol:

foo_KCONF=CONFIG_FOO

This file will be further loaded by the Makefile as a
configuration file: all foo_KCONF will be considered variables
and all CONFIG_FOO will be their associate values. The suffix
is added to be able to differentiate the new variables from
all existing variables in the Makefile system. Each kernel
module will receive its CONFIG_FOO option as macro in the
compilation command (-D parameter) and will expose it along
with other module attributes (attributes of struct module).

Note: this patch is built on the assumption that each driver
has exactly one associate CONFIG_FOO symbol. CONFIG_* options
that are required by CONFIG_FOO are not included. They can be
found by SAT resolvers.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/scripts/kconfig/streamline_config.pl b/scripts/kconfig/streamline_config.pl
index b8c7b29..686d02a 100755
--- a/scripts/kconfig/streamline_config.pl
+++ b/scripts/kconfig/streamline_config.pl
@@ -128,9 +128,11 @@ my @config_file = read_config;
 # Parse options
 my $localmodconfig = 0;
 my $localyesconfig = 0;
+my $genmoduleksymb = 0;
 
 GetOptions("localmodconfig" => \$localmodconfig,
-	   "localyesconfig" => \$localyesconfig);
+	   "localyesconfig" => \$localyesconfig,
+	   "genmoduleksymb" => \$genmoduleksymb);
 
 # Get the build source and top level Kconfig file (passed in)
 my $ksource = ($ARGV[0] ? $ARGV[0] : '.');
@@ -147,6 +149,7 @@ my %objects;
 my $var;
 my $iflevel = 0;
 my @ifdeps;
+my @drv_objs;
 
 # prevent recursion
 my %read_kconfigs;
@@ -348,6 +351,31 @@ foreach my $makefile (@makefiles) {
     close($infile);
 }
 
+foreach my $obj_key ( keys %objects )
+{
+	my @config_options = @{$objects{$obj_key}};
+	# Last index of array is 0 is equivalent to array's size is 1
+	if ( $#config_options == 0) {
+		push(@drv_objs, $obj_key);
+	}
+}
+
+sub gen_module_kconfigs {
+	my $module_ksymb = $ENV{'srctree'}."/scripts/Module.ksymb";
+
+	open(my $ksymbfile, '>', $module_ksymb) || die "Can not open $module_ksymb for writing";
+
+	foreach (@drv_objs) {
+		print $ksymbfile "$_" . "_KCONF=" . "@{$objects{$_}}\n";
+	}
+	close($ksymbfile);
+}
+
+if ($genmoduleksymb) {
+	gen_module_kconfigs();
+	exit(0);
+}
+
 my %modules;
 my $linfile;
 
-- 
2.7.4

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
> Add generation of ./scripts/Module.ksymb file containing
> associations of driver file names and corresponding CONFIG_*
> symbol:
> 
> foo_KCONF=CONFIG_FOO
> 
> This file will be further loaded by the Makefile as a
> configuration file: all foo_KCONF will be considered variables
> and all CONFIG_FOO will be their associate values.

Will the file generated always work? What are the current
restrictions on kconfig variable names?

> The suffix
> is added to be able to differentiate the new variables from
> all existing variables in the Makefile system. Each kernel
> module will receive its CONFIG_FOO option as macro in the
> compilation command (-D parameter) and will expose it along
> with other module attributes (attributes of struct module).

This is done later so this description does not belong here.
You want to describe what this patch does alone, if it has
potential for the future just mention basics, and that will will
be done later.

> Note: this patch is built on the assumption that each driver
> has exactly one associate CONFIG_FOO symbol.

This should note be a note, it should be an important aspect of
your changes -- you should try to describe when a direct mapping
is possible and when it is not.

> CONFIG_* options
> that are required by CONFIG_FOO are not included. They can be
> found by SAT resolvers.

You can leave this out.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> 
> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>  1 file changed, 29 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/kconfig/streamline_config.pl b/scripts/kconfig/streamline_config.pl
> index b8c7b29..686d02a 100755
> --- a/scripts/kconfig/streamline_config.pl
> +++ b/scripts/kconfig/streamline_config.pl
> @@ -128,9 +128,11 @@ my @config_file = read_config;
>  # Parse options
>  my $localmodconfig = 0;
>  my $localyesconfig = 0;
> +my $genmoduleksymb = 0;
>  
>  GetOptions("localmodconfig" => \$localmodconfig,
> -	   "localyesconfig" => \$localyesconfig);
> +	   "localyesconfig" => \$localyesconfig,
> +	   "genmoduleksymb" => \$genmoduleksymb);
>  
>  # Get the build source and top level Kconfig file (passed in)
>  my $ksource = ($ARGV[0] ? $ARGV[0] : '.');
> @@ -147,6 +149,7 @@ my %objects;
>  my $var;
>  my $iflevel = 0;
>  my @ifdeps;
> +my @drv_objs;
>  
>  # prevent recursion
>  my %read_kconfigs;
> @@ -348,6 +351,31 @@ foreach my $makefile (@makefiles) {
>      close($infile);
>  }
>  
> +foreach my $obj_key ( keys %objects )
> +{
> +	my @config_options = @{$objects{$obj_key}};
> +	# Last index of array is 0 is equivalent to array's size is 1
> +	if ( $#config_options == 0) {
> +		push(@drv_objs, $obj_key);
> +	}
> +}
> +
> +sub gen_module_kconfigs {
> +	my $module_ksymb = $ENV{'srctree'}."/scripts/Module.ksymb";
> +
> +	open(my $ksymbfile, '>', $module_ksymb) || die "Can not open $module_ksymb for writing";
> +
> +	foreach (@drv_objs) {
> +		print $ksymbfile "$_" . "_KCONF=" . "@{$objects{$_}}\n";
> +	}
> +	close($ksymbfile);
> +}
> +
> +if ($genmoduleksymb) {
> +	gen_module_kconfigs();
> +	exit(0);
> +}
> +
>  my %modules;
>  my $linfile;

This is nice for experimentation, however as you note we want
to evaluate a better way to do this. While at it, can you describe
what algorithm currently is used by scripts/kconfig/streamline_config.pl
to collect %objects and the associated CONFIG symbol ? Can you think
of possible flaws to it ?

The generation here is also forced, we want to enable this to be optional
so please consider adding a Kconfig entry for this as a feature, just as
the module versioning stuff has one.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
> Add generation of ./scripts/Module.ksymb file containing
> associations of driver file names and corresponding CONFIG_*
> symbol:
> 
> foo_KCONF=CONFIG_FOO
> 
> This file will be further loaded by the Makefile as a
> configuration file: all foo_KCONF will be considered variables
> and all CONFIG_FOO will be their associate values.

Will the file generated always work? What are the current
restrictions on kconfig variable names?

> The suffix
> is added to be able to differentiate the new variables from
> all existing variables in the Makefile system. Each kernel
> module will receive its CONFIG_FOO option as macro in the
> compilation command (-D parameter) and will expose it along
> with other module attributes (attributes of struct module).

This is done later so this description does not belong here.
You want to describe what this patch does alone, if it has
potential for the future just mention basics, and that will will
be done later.

> Note: this patch is built on the assumption that each driver
> has exactly one associate CONFIG_FOO symbol.

This should note be a note, it should be an important aspect of
your changes -- you should try to describe when a direct mapping
is possible and when it is not.

> CONFIG_* options
> that are required by CONFIG_FOO are not included. They can be
> found by SAT resolvers.

You can leave this out.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> 
> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>  1 file changed, 29 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/kconfig/streamline_config.pl b/scripts/kconfig/streamline_config.pl
> index b8c7b29..686d02a 100755
> --- a/scripts/kconfig/streamline_config.pl
> +++ b/scripts/kconfig/streamline_config.pl
> @@ -128,9 +128,11 @@ my @config_file = read_config;
>  # Parse options
>  my $localmodconfig = 0;
>  my $localyesconfig = 0;
> +my $genmoduleksymb = 0;
>  
>  GetOptions("localmodconfig" => \$localmodconfig,
> -	   "localyesconfig" => \$localyesconfig);
> +	   "localyesconfig" => \$localyesconfig,
> +	   "genmoduleksymb" => \$genmoduleksymb);
>  
>  # Get the build source and top level Kconfig file (passed in)
>  my $ksource = ($ARGV[0] ? $ARGV[0] : '.');
> @@ -147,6 +149,7 @@ my %objects;
>  my $var;
>  my $iflevel = 0;
>  my @ifdeps;
> +my @drv_objs;
>  
>  # prevent recursion
>  my %read_kconfigs;
> @@ -348,6 +351,31 @@ foreach my $makefile (@makefiles) {
>      close($infile);
>  }
>  
> +foreach my $obj_key ( keys %objects )
> +{
> +	my @config_options = @{$objects{$obj_key}};
> +	# Last index of array is 0 is equivalent to array's size is 1
> +	if ( $#config_options == 0) {
> +		push(@drv_objs, $obj_key);
> +	}
> +}
> +
> +sub gen_module_kconfigs {
> +	my $module_ksymb = $ENV{'srctree'}."/scripts/Module.ksymb";
> +
> +	open(my $ksymbfile, '>', $module_ksymb) || die "Can not open $module_ksymb for writing";
> +
> +	foreach (@drv_objs) {
> +		print $ksymbfile "$_" . "_KCONF=" . "@{$objects{$_}}\n";
> +	}
> +	close($ksymbfile);
> +}
> +
> +if ($genmoduleksymb) {
> +	gen_module_kconfigs();
> +	exit(0);
> +}
> +
>  my %modules;
>  my $linfile;

This is nice for experimentation, however as you note we want
to evaluate a better way to do this. While at it, can you describe
what algorithm currently is used by scripts/kconfig/streamline_config.pl
to collect %objects and the associated CONFIG symbol ? Can you think
of possible flaws to it ?

The generation here is also forced, we want to enable this to be optional
so please consider adding a Kconfig entry for this as a feature, just as
the module versioning stuff has one.

  Luis

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Luis R. Rodriguez]

Also I think your subject has a typo.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Luis R. Rodriguez]

Also I think your subject has a typo.

  Luis

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Cristina-Gabriela Moraru]

2016-08-20 15:59 GMT+02:00 Cristina-Gabriela Moraru
<cristina.moraru09@gmail.com>:
>
>
> 2016-08-18 20:22 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
>>
>> On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
>> > Add generation of ./scripts/Module.ksymb file containing
>> > associations of driver file names and corresponding CONFIG_*
>> > symbol:
>> >
>> > foo_KCONF=3DCONFIG_FOO
>> >
>> > This file will be further loaded by the Makefile as a
>> > configuration file: all foo_KCONF will be considered variables
>> > and all CONFIG_FOO will be their associate values.
>>
>> Will the file generated always work? What are the current
>> restrictions on kconfig variable names?
>>
>
> The file generated will always work. It's loaded with makefile's "include=
"
> directive which is quite straightforward.
>
> If you're referring to  CONFIG_* variables, I can't find any restriction
> regarding size or anything else.
> If you're referring to the newly created makefile variable foo_KCONF I on=
ly
> found this in the documentation:
> "A variable name may be any sequence of characters not containing =E2=80=
=98:=E2=80=99, =E2=80=98#=E2=80=99,
> =E2=80=98=3D=E2=80=99, or whitespace."
> "It is traditional to use upper case letters in variable names, but we
> recommend using lower case letters for variable names that serve internal
> purposes in the makefile, and reserving upper case for parameters that
> control implicit rules or for parameters that the user should override wi=
th
> command options"
>
> Since the module name is lowercase I put the suffix uppercase in order to=
 be
> more visible.
>
>>
>> > The suffix
>> > is added to be able to differentiate the new variables from
>> > all existing variables in the Makefile system. Each kernel
>> > module will receive its CONFIG_FOO option as macro in the
>> > compilation command (-D parameter) and will expose it along
>> > with other module attributes (attributes of struct module).
>>
>> This is done later so this description does not belong here.
>> You want to describe what this patch does alone, if it has
>> potential for the future just mention basics, and that will will
>> be done later.
>>
>> > Note: this patch is built on the assumption that each driver
>> > has exactly one associate CONFIG_FOO symbol.
>>
>> This should note be a note, it should be an important aspect of
>> your changes -- you should try to describe when a direct mapping
>> is possible and when it is not.
>>
>> > CONFIG_* options
>> > that are required by CONFIG_FOO are not included. They can be
>> > found by SAT resolvers.
>>
>> You can leave this out.
>>
>> > This patch is part of a research project within
>> > Google Summer of Code of porting 'make localmodconfig'
>> > for backported drivers. The goal is to enable each
>> > module to expose in /sys its corresponding CONFIG_* option.
>> > The value of this attribute will be dynamically pegged by
>> > modpost without requiring extra work from the driver developers.
>> > Further, this information will be used by a hardware interogation
>> > tool to extract build information about the existing devices.
>>
>> You can leave this out.
>>
>> >
>> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
>> > ---
>> >  scripts/kconfig/streamline_config.pl | 30
>> > +++++++++++++++++++++++++++++-
>> >  1 file changed, 29 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/scripts/kconfig/streamline_config.pl
>> > b/scripts/kconfig/streamline_config.pl
>> > index b8c7b29..686d02a 100755
>> > --- a/scripts/kconfig/streamline_config.pl
>> > +++ b/scripts/kconfig/streamline_config.pl
>> > @@ -128,9 +128,11 @@ my @config_file =3D read_config;
>> >  # Parse options
>> >  my $localmodconfig =3D 0;
>> >  my $localyesconfig =3D 0;
>> > +my $genmoduleksymb =3D 0;
>> >
>> >  GetOptions("localmodconfig" =3D> \$localmodconfig,
>> > -        "localyesconfig" =3D> \$localyesconfig);
>> > +        "localyesconfig" =3D> \$localyesconfig,
>> > +        "genmoduleksymb" =3D> \$genmoduleksymb);
>> >
>> >  # Get the build source and top level Kconfig file (passed in)
>> >  my $ksource =3D ($ARGV[0] ? $ARGV[0] : '.');
>> > @@ -147,6 +149,7 @@ my %objects;
>> >  my $var;
>> >  my $iflevel =3D 0;
>> >  my @ifdeps;
>> > +my @drv_objs;
>> >
>> >  # prevent recursion
>> >  my %read_kconfigs;
>> > @@ -348,6 +351,31 @@ foreach my $makefile (@makefiles) {
>> >      close($infile);
>> >  }
>> >
>> > +foreach my $obj_key ( keys %objects )
>> > +{
>> > +     my @config_options =3D @{$objects{$obj_key}};
>> > +     # Last index of array is 0 is equivalent to array's size is 1
>> > +     if ( $#config_options =3D=3D 0) {
>> > +             push(@drv_objs, $obj_key);
>> > +     }
>> > +}
>> > +
>> > +sub gen_module_kconfigs {
>> > +     my $module_ksymb =3D $ENV{'srctree'}."/scripts/Module.ksymb";
>> > +
>> > +     open(my $ksymbfile, '>', $module_ksymb) || die "Can not open
>> > $module_ksymb for writing";
>> > +
>> > +     foreach (@drv_objs) {
>> > +             print $ksymbfile "$_" . "_KCONF=3D" . "@{$objects{$_}}\n=
";
>> > +     }
>> > +     close($ksymbfile);
>> > +}
>> > +
>> > +if ($genmoduleksymb) {
>> > +     gen_module_kconfigs();
>> > +     exit(0);
>> > +}
>> > +
>> >  my %modules;
>> >  my $linfile;
>>
>> This is nice for experimentation, however as you note we want
>> to evaluate a better way to do this. While at it, can you describe
>> what algorithm currently is used by scripts/kconfig/streamline_config.pl
>> to collect %objects and the associated CONFIG symbol ? Can you think
>> of possible flaws to it ?
>>
>
> Shortly, all makefiles are parsed searching for patterns such as:
>
> obj-$(CONFIG_FOO) :=3D obj-foo1.o obj-foo2.o ... etc.
>
> Each obj-foo is added in the hashmap as key with the value
> array(CONFIG_FOO).
> If obj-foo already exists in the hashmap CONFIG_FOO is appended to the ar=
ray
> value.
>
> I noted that there are situations where an object obj-foo has as associat=
es
> more identical CONFIG_FOOs.
> Since I select only the names which have exaclty one CONFIG_* associated
> this is a bit confusing. I am missing one category of drivers which can b=
e
> found:
>
> driver-foo ---- CONFIG_FOO CONFIG_FOO
>
> For example: nfit CONFIG_ACPI_NFIT CONFIG_ACPI_NFIT -- this actually resu=
lts
> in  a module in /sys which does not have a content into kconfig_ksymb
> although it has exactly one match.
>
> Another thing I can think of is that sometimes we find lines such as:
>
> foo-name-$(CONFIG_FOO) =3D obj-foo1.o obj-foo2.o obj-foo3.o
>
> This can be a clue that the final kernel module is named foo-name and cou=
ld
> solve some ambiguous cases.
> This are just a few situations I can think of to be kept in mind for the
> next Module.ksymb generator
>
>>
>> The generation here is also forced, we want to enable this to be optiona=
l
>> so please consider adding a Kconfig entry for this as a feature, just as
>> the module versioning stuff has one.
>
>
> OK
>>
>>
>>   Luis
>
>

Message forwarded since original message was rejected by lkml and
backports for being HTML format.
Cristina
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Cristina-Gabriela Moraru]

2016-08-20 15:59 GMT+02:00 Cristina-Gabriela Moraru
<cristina.moraru09@gmail.com>:
>
>
> 2016-08-18 20:22 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
>>
>> On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
>> > Add generation of ./scripts/Module.ksymb file containing
>> > associations of driver file names and corresponding CONFIG_*
>> > symbol:
>> >
>> > foo_KCONF=CONFIG_FOO
>> >
>> > This file will be further loaded by the Makefile as a
>> > configuration file: all foo_KCONF will be considered variables
>> > and all CONFIG_FOO will be their associate values.
>>
>> Will the file generated always work? What are the current
>> restrictions on kconfig variable names?
>>
>
> The file generated will always work. It's loaded with makefile's "include"
> directive which is quite straightforward.
>
> If you're referring to  CONFIG_* variables, I can't find any restriction
> regarding size or anything else.
> If you're referring to the newly created makefile variable foo_KCONF I only
> found this in the documentation:
> "A variable name may be any sequence of characters not containing ‘:’, ‘#’,
> ‘=’, or whitespace."
> "It is traditional to use upper case letters in variable names, but we
> recommend using lower case letters for variable names that serve internal
> purposes in the makefile, and reserving upper case for parameters that
> control implicit rules or for parameters that the user should override with
> command options"
>
> Since the module name is lowercase I put the suffix uppercase in order to be
> more visible.
>
>>
>> > The suffix
>> > is added to be able to differentiate the new variables from
>> > all existing variables in the Makefile system. Each kernel
>> > module will receive its CONFIG_FOO option as macro in the
>> > compilation command (-D parameter) and will expose it along
>> > with other module attributes (attributes of struct module).
>>
>> This is done later so this description does not belong here.
>> You want to describe what this patch does alone, if it has
>> potential for the future just mention basics, and that will will
>> be done later.
>>
>> > Note: this patch is built on the assumption that each driver
>> > has exactly one associate CONFIG_FOO symbol.
>>
>> This should note be a note, it should be an important aspect of
>> your changes -- you should try to describe when a direct mapping
>> is possible and when it is not.
>>
>> > CONFIG_* options
>> > that are required by CONFIG_FOO are not included. They can be
>> > found by SAT resolvers.
>>
>> You can leave this out.
>>
>> > This patch is part of a research project within
>> > Google Summer of Code of porting 'make localmodconfig'
>> > for backported drivers. The goal is to enable each
>> > module to expose in /sys its corresponding CONFIG_* option.
>> > The value of this attribute will be dynamically pegged by
>> > modpost without requiring extra work from the driver developers.
>> > Further, this information will be used by a hardware interogation
>> > tool to extract build information about the existing devices.
>>
>> You can leave this out.
>>
>> >
>> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
>> > ---
>> >  scripts/kconfig/streamline_config.pl | 30
>> > +++++++++++++++++++++++++++++-
>> >  1 file changed, 29 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/scripts/kconfig/streamline_config.pl
>> > b/scripts/kconfig/streamline_config.pl
>> > index b8c7b29..686d02a 100755
>> > --- a/scripts/kconfig/streamline_config.pl
>> > +++ b/scripts/kconfig/streamline_config.pl
>> > @@ -128,9 +128,11 @@ my @config_file = read_config;
>> >  # Parse options
>> >  my $localmodconfig = 0;
>> >  my $localyesconfig = 0;
>> > +my $genmoduleksymb = 0;
>> >
>> >  GetOptions("localmodconfig" => \$localmodconfig,
>> > -        "localyesconfig" => \$localyesconfig);
>> > +        "localyesconfig" => \$localyesconfig,
>> > +        "genmoduleksymb" => \$genmoduleksymb);
>> >
>> >  # Get the build source and top level Kconfig file (passed in)
>> >  my $ksource = ($ARGV[0] ? $ARGV[0] : '.');
>> > @@ -147,6 +149,7 @@ my %objects;
>> >  my $var;
>> >  my $iflevel = 0;
>> >  my @ifdeps;
>> > +my @drv_objs;
>> >
>> >  # prevent recursion
>> >  my %read_kconfigs;
>> > @@ -348,6 +351,31 @@ foreach my $makefile (@makefiles) {
>> >      close($infile);
>> >  }
>> >
>> > +foreach my $obj_key ( keys %objects )
>> > +{
>> > +     my @config_options = @{$objects{$obj_key}};
>> > +     # Last index of array is 0 is equivalent to array's size is 1
>> > +     if ( $#config_options == 0) {
>> > +             push(@drv_objs, $obj_key);
>> > +     }
>> > +}
>> > +
>> > +sub gen_module_kconfigs {
>> > +     my $module_ksymb = $ENV{'srctree'}."/scripts/Module.ksymb";
>> > +
>> > +     open(my $ksymbfile, '>', $module_ksymb) || die "Can not open
>> > $module_ksymb for writing";
>> > +
>> > +     foreach (@drv_objs) {
>> > +             print $ksymbfile "$_" . "_KCONF=" . "@{$objects{$_}}\n";
>> > +     }
>> > +     close($ksymbfile);
>> > +}
>> > +
>> > +if ($genmoduleksymb) {
>> > +     gen_module_kconfigs();
>> > +     exit(0);
>> > +}
>> > +
>> >  my %modules;
>> >  my $linfile;
>>
>> This is nice for experimentation, however as you note we want
>> to evaluate a better way to do this. While at it, can you describe
>> what algorithm currently is used by scripts/kconfig/streamline_config.pl
>> to collect %objects and the associated CONFIG symbol ? Can you think
>> of possible flaws to it ?
>>
>
> Shortly, all makefiles are parsed searching for patterns such as:
>
> obj-$(CONFIG_FOO) := obj-foo1.o obj-foo2.o ... etc.
>
> Each obj-foo is added in the hashmap as key with the value
> array(CONFIG_FOO).
> If obj-foo already exists in the hashmap CONFIG_FOO is appended to the array
> value.
>
> I noted that there are situations where an object obj-foo has as associates
> more identical CONFIG_FOOs.
> Since I select only the names which have exaclty one CONFIG_* associated
> this is a bit confusing. I am missing one category of drivers which can be
> found:
>
> driver-foo ---- CONFIG_FOO CONFIG_FOO
>
> For example: nfit CONFIG_ACPI_NFIT CONFIG_ACPI_NFIT -- this actually results
> in  a module in /sys which does not have a content into kconfig_ksymb
> although it has exactly one match.
>
> Another thing I can think of is that sometimes we find lines such as:
>
> foo-name-$(CONFIG_FOO) = obj-foo1.o obj-foo2.o obj-foo3.o
>
> This can be a clue that the final kernel module is named foo-name and could
> solve some ambiguous cases.
> This are just a few situations I can think of to be kept in mind for the
> next Module.ksymb generator
>
>>
>> The generation here is also forced, we want to enable this to be optional
>> so please consider adding a Kconfig entry for this as a feature, just as
>> the module versioning stuff has one.
>
>
> OK
>>
>>
>>   Luis
>
>

Message forwarded since original message was rejected by lkml and
backports for being HTML format.
Cristina

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Luis R. Rodriguez]

On Sat, Aug 20, 2016 at 03:59:17PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:22 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> 
> > On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
> > > Add generation of ./scripts/Module.ksymb file containing
> > > associations of driver file names and corresponding CONFIG_*
> > > symbol:
> > >
> > > foo_KCONF=CONFIG_FOO
> > >
> > > This file will be further loaded by the Makefile as a
> > > configuration file: all foo_KCONF will be considered variables
> > > and all CONFIG_FOO will be their associate values.
> >
> > Will the file generated always work? What are the current
> > restrictions on kconfig variable names?
> >
> >
> The file generated will always work. It's loaded with makefile's "include"
> directive which is quite straightforward.
> 
> If you're referring to  CONFIG_* variables, I can't find any restriction
> regarding size or anything else.
> If you're referring to the newly created makefile variable foo_KCONF I only
> found this in the documentation:
> "A variable name may be any sequence of characters not containing ‘:’, ‘#’,
> ‘=’, or whitespace."

Both since we'd up with a file with tons of entries like:

bar_KCONF=CONFIG_BAR
foo_KCONF=CONFIG_FOO

I was asking really what checks do we have to ensure either side in these lines
will not make a Makefile barf. For instance using "\" will surely barf. Does
Kconfig check for sanity ? If so then what are the rules for Kconfig variables?
Makefiles then have the above description you mentioned, and I was curious how
and if we want to vet for correctness for both.

> "It is traditional to use upper case letters in variable names, but we
> recommend using lower case letters for variable names that serve internal
> purposes in the makefile, and reserving upper case for parameters that
> control implicit rules or for parameters that the user should override with
> command options"
> 
> Since the module name is lowercase I put the suffix uppercase in order to
> be more visible.

OK thanks.

> > This is nice for experimentation, however as you note we want
> > to evaluate a better way to do this. While at it, can you describe
> > what algorithm currently is used by scripts/kconfig/streamline_config.pl
> > to collect %objects and the associated CONFIG symbol ? Can you think
> > of possible flaws to it ?
> >
> >
> Shortly, all makefiles are parsed searching for patterns such as:
> 
> obj-$(CONFIG_FOO) := obj-foo1.o obj-foo2.o ... etc.
> 
> Each obj-foo is added in the hashmap as key with the value
> array(CONFIG_FOO).
> If obj-foo already exists in the hashmap CONFIG_FOO is appended to the
> array value.

So this fails to then capture lib-$(CONFIG_BAR) then. These can be modules
too. So we'd need something to loop over all possible valid targets to be
able to easily extend this with future targets, in case we get something
other than obj- and lib-. For instance I am looking to add a force-obj-y
and force-lib-y, later this may mean supporting force-obj-m and foce-lib-m,
the actual name however is still up in the air [0], however my point is
we want to be easily able to extend the series of targets used for modules.

This may prove useful to the build something for other things.

Also, does the scripts/kconfig/streamline_config.pl algorithm support
where you have more than one line on a target, for instance:

obj-$(CONFIG_FOO) := foo1.o \
		     foo2.o \
		     foo3.o

If not we'd need to figure out a way to capture these then?

[0] https://lkml.kernel.org/r/20160822235910.GW3296@wotan.suse.de

> I noted that there are situations where an object obj-foo has as associates
> more identical CONFIG_FOOs.
> Since I select only the names which have exaclty one CONFIG_* associated
> this is a bit confusing. I am missing one category of drivers which can be
> found:
> 
> driver-foo ---- CONFIG_FOO CONFIG_FOO

What do these 3 columns represent here ?

> For example: nfit CONFIG_ACPI_NFIT CONFIG_ACPI_NFIT -- this actually
> results in  a module in /sys which does not have a content into
> kconfig_ksymb although it has exactly one match.
> 
> Another thing I can think of is that sometimes we find lines such as:
> 
> foo-name-$(CONFIG_FOO) = obj-foo1.o obj-foo2.o obj-foo3.o
> 
> This can be a clue that the final kernel module is named foo-name and could
> solve some ambiguous cases.
> This are just a few situations I can think of to be kept in mind for the
> next Module.ksymb generator

Indeed thanks, I think it'd be nice if we can iron all of them out. If we cannot
then we should annotate why we can't and see if we are willing to compromise
to help address this.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 1/5] Add generation of Module.symb in streamline_config

[Luis R. Rodriguez]

On Sat, Aug 20, 2016 at 03:59:17PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:22 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> 
> > On Wed, Aug 17, 2016 at 09:26:59PM +0200, Cristina Moraru wrote:
> > > Add generation of ./scripts/Module.ksymb file containing
> > > associations of driver file names and corresponding CONFIG_*
> > > symbol:
> > >
> > > foo_KCONF=CONFIG_FOO
> > >
> > > This file will be further loaded by the Makefile as a
> > > configuration file: all foo_KCONF will be considered variables
> > > and all CONFIG_FOO will be their associate values.
> >
> > Will the file generated always work? What are the current
> > restrictions on kconfig variable names?
> >
> >
> The file generated will always work. It's loaded with makefile's "include"
> directive which is quite straightforward.
> 
> If you're referring to  CONFIG_* variables, I can't find any restriction
> regarding size or anything else.
> If you're referring to the newly created makefile variable foo_KCONF I only
> found this in the documentation:
> "A variable name may be any sequence of characters not containing ‘:’, ‘#’,
> ‘=’, or whitespace."

Both since we'd up with a file with tons of entries like:

bar_KCONF=CONFIG_BAR
foo_KCONF=CONFIG_FOO

I was asking really what checks do we have to ensure either side in these lines
will not make a Makefile barf. For instance using "\" will surely barf. Does
Kconfig check for sanity ? If so then what are the rules for Kconfig variables?
Makefiles then have the above description you mentioned, and I was curious how
and if we want to vet for correctness for both.

> "It is traditional to use upper case letters in variable names, but we
> recommend using lower case letters for variable names that serve internal
> purposes in the makefile, and reserving upper case for parameters that
> control implicit rules or for parameters that the user should override with
> command options"
> 
> Since the module name is lowercase I put the suffix uppercase in order to
> be more visible.

OK thanks.

> > This is nice for experimentation, however as you note we want
> > to evaluate a better way to do this. While at it, can you describe
> > what algorithm currently is used by scripts/kconfig/streamline_config.pl
> > to collect %objects and the associated CONFIG symbol ? Can you think
> > of possible flaws to it ?
> >
> >
> Shortly, all makefiles are parsed searching for patterns such as:
> 
> obj-$(CONFIG_FOO) := obj-foo1.o obj-foo2.o ... etc.
> 
> Each obj-foo is added in the hashmap as key with the value
> array(CONFIG_FOO).
> If obj-foo already exists in the hashmap CONFIG_FOO is appended to the
> array value.

So this fails to then capture lib-$(CONFIG_BAR) then. These can be modules
too. So we'd need something to loop over all possible valid targets to be
able to easily extend this with future targets, in case we get something
other than obj- and lib-. For instance I am looking to add a force-obj-y
and force-lib-y, later this may mean supporting force-obj-m and foce-lib-m,
the actual name however is still up in the air [0], however my point is
we want to be easily able to extend the series of targets used for modules.

This may prove useful to the build something for other things.

Also, does the scripts/kconfig/streamline_config.pl algorithm support
where you have more than one line on a target, for instance:

obj-$(CONFIG_FOO) := foo1.o \
		     foo2.o \
		     foo3.o

If not we'd need to figure out a way to capture these then?

[0] https://lkml.kernel.org/r/20160822235910.GW3296@wotan.suse.de

> I noted that there are situations where an object obj-foo has as associates
> more identical CONFIG_FOOs.
> Since I select only the names which have exaclty one CONFIG_* associated
> this is a bit confusing. I am missing one category of drivers which can be
> found:
> 
> driver-foo ---- CONFIG_FOO CONFIG_FOO

What do these 3 columns represent here ?

> For example: nfit CONFIG_ACPI_NFIT CONFIG_ACPI_NFIT -- this actually
> results in  a module in /sys which does not have a content into
> kconfig_ksymb although it has exactly one match.
> 
> Another thing I can think of is that sometimes we find lines such as:
> 
> foo-name-$(CONFIG_FOO) = obj-foo1.o obj-foo2.o obj-foo3.o
> 
> This can be a clue that the final kernel module is named foo-name and could
> solve some ambiguous cases.
> This are just a few situations I can think of to be kept in mind for the
> next Module.ksymb generator

Indeed thanks, I think it'd be nice if we can iron all of them out. If we cannot
then we should annotate why we can't and see if we are willing to compromise
to help address this.

  Luis

[RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Cristina Moraru]

Add CONFIG symbol to kernel modules as a define via -D
compilation parameter. The CONFIG_FOO symbol for each
module is determined by the module name, using the
associations from Module.ksymb. This file is loaded
using the 'include' directive, thus run like a regular
makefile. The format of the content of the file is the
following:

foo_KCONF=CONFIG_FOO

creating a set of Makefile variables foo_KCONF with the
CONFIG_FOO as values.

The Makefile then adds this value in the compilation
command with -DKBUILD_KCONF='"CONFIG_FOO"'.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 scripts/Makefile.lib | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index e7df0f5..8ae9b7f 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
 subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
 obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
 
+# Include Module.ksymb which contains the associations of modules' names
+# and corresponding CONFIG_* options
+include Module.ksymb
+
 # These flags are needed for modversions and compiling, so we define them here
 # already
 # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
@@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
 basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
 modname_flags  = $(if $(filter 1,$(words $(modname))),\
                  -DKBUILD_MODNAME=$(call name-fix,$(modname)))
+ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
+ksymb_flags = $(if $(filter 1,$(words $(modname))),\
+                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
 
 orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
                  $(ccflags-y) $(CFLAGS_$(basetarget).o)
@@ -162,7 +169,7 @@ endif
 
 c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__c_flags) $(modkern_cflags)                           \
-		 $(basename_flags) $(modname_flags)
+		 $(basename_flags) $(modname_flags) $(ksymb_flags)
 
 a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__a_flags) $(modkern_aflags)
-- 
2.7.4

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> Add  CONFIG symbol to kernel modules as a define via -D

Perhaps better worded as:

When modules have a direct Kconfig CONFIG_ symbol associated with
we want to be able to make it available to the build system when we
are building the module.

You can then describe you do this with -D.

> compilation parameter. The CONFIG_FOO symbol for each
> module is determined by the module name, using the
> associations from Module.ksymb. This file is loaded
> using the 'include' directive, thus run like a regular
> makefile. The format of the content of the file is the
> following:
> 
> foo_KCONF=CONFIG_FOO
> 
> creating a set of Makefile variables foo_KCONF with the
> CONFIG_FOO as values.
> 
> The Makefile then adds this value in the compilation
> command with -DKBUILD_KCONF='"CONFIG_FOO"'.

Very nice. What would it mean if it lacks this ?
This should be explained on the commit log.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out as its part of the cover letter and
not relevant to the patch. You can however mention you will
use this later to make the define more useful in userspace.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/Makefile.lib | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index e7df0f5..8ae9b7f 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
>  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
>  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
>  
> +# Include Module.ksymb which contains the associations of modules' names
> +# and corresponding CONFIG_* options
> +include Module.ksymb

Is this file always generated? If not prefixing the include call with - would
be better. If we are going to make this optional, then an ifdef wrapper would
suffice as we know it'd be expected only if the new feature was enabled.

> +
>  # These flags are needed for modversions and compiling, so we define them here
>  # already
>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))

Are clashes possible with this formula? Can we end up with two results for instance?
If not what prevents current konfig logic and namespace from a clash ? If we do
not have anything to prevent a clash, what can we do to help make such clash not
possible ?

>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
> @@ -162,7 +169,7 @@ endif
>  
>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__c_flags) $(modkern_cflags)                           \
> -		 $(basename_flags) $(modname_flags)
> +		 $(basename_flags) $(modname_flags) $(ksymb_flags)
>  
>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__a_flags) $(modkern_aflags)
> -- 

Other than that looks good!

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> Add  CONFIG symbol to kernel modules as a define via -D

Perhaps better worded as:

When modules have a direct Kconfig CONFIG_ symbol associated with
we want to be able to make it available to the build system when we
are building the module.

You can then describe you do this with -D.

> compilation parameter. The CONFIG_FOO symbol for each
> module is determined by the module name, using the
> associations from Module.ksymb. This file is loaded
> using the 'include' directive, thus run like a regular
> makefile. The format of the content of the file is the
> following:
> 
> foo_KCONF=CONFIG_FOO
> 
> creating a set of Makefile variables foo_KCONF with the
> CONFIG_FOO as values.
> 
> The Makefile then adds this value in the compilation
> command with -DKBUILD_KCONF='"CONFIG_FOO"'.

Very nice. What would it mean if it lacks this ?
This should be explained on the commit log.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out as its part of the cover letter and
not relevant to the patch. You can however mention you will
use this later to make the define more useful in userspace.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/Makefile.lib | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index e7df0f5..8ae9b7f 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
>  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
>  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
>  
> +# Include Module.ksymb which contains the associations of modules' names
> +# and corresponding CONFIG_* options
> +include Module.ksymb

Is this file always generated? If not prefixing the include call with - would
be better. If we are going to make this optional, then an ifdef wrapper would
suffice as we know it'd be expected only if the new feature was enabled.

> +
>  # These flags are needed for modversions and compiling, so we define them here
>  # already
>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))

Are clashes possible with this formula? Can we end up with two results for instance?
If not what prevents current konfig logic and namespace from a clash ? If we do
not have anything to prevent a clash, what can we do to help make such clash not
possible ?

>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
> @@ -162,7 +169,7 @@ endif
>  
>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__c_flags) $(modkern_cflags)                           \
> -		 $(basename_flags) $(modname_flags)
> +		 $(basename_flags) $(modname_flags) $(ksymb_flags)
>  
>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__a_flags) $(modkern_aflags)
> -- 

Other than that looks good!

  Luis

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Luis R. Rodriguez]

On Thu, Aug 18, 2016 at 08:10:07PM +0200, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> 
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/Makefile.lib | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> > index e7df0f5..8ae9b7f 100644
> > --- a/scripts/Makefile.lib
> > +++ b/scripts/Makefile.lib
> > @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
> >  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
> >  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
> >  
> > +# Include Module.ksymb which contains the associations of modules' names
> > +# and corresponding CONFIG_* options
> > +include Module.ksymb
> 
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

The ordering of the patches 2 and 3 seem to be backward, you should first
generate the file otherwise this patch will break the build as the file
is not present.

> > +
> >  # These flags are needed for modversions and compiling, so we define them here
> >  # already
> >  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> > @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> > +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> > +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> > +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> 
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

To help test this I've added your patches to a development branch on my linux-next
tree to see if 0-day picks up any issues.

Guenter, to be even more thorough can I trouble you for a spin of this branch?

https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160818-gsoc-kconf_symb

Results should help give us an idea of troubling areas if this were to be added
as a new Kconfig feature even for R&D.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Luis R. Rodriguez]

On Thu, Aug 18, 2016 at 08:10:07PM +0200, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> 
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/Makefile.lib | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> > index e7df0f5..8ae9b7f 100644
> > --- a/scripts/Makefile.lib
> > +++ b/scripts/Makefile.lib
> > @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
> >  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
> >  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
> >  
> > +# Include Module.ksymb which contains the associations of modules' names
> > +# and corresponding CONFIG_* options
> > +include Module.ksymb
> 
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

The ordering of the patches 2 and 3 seem to be backward, you should first
generate the file otherwise this patch will break the build as the file
is not present.

> > +
> >  # These flags are needed for modversions and compiling, so we define them here
> >  # already
> >  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> > @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> > +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> > +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> > +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> 
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

To help test this I've added your patches to a development branch on my linux-next
tree to see if 0-day picks up any issues.

Guenter, to be even more thorough can I trouble you for a spin of this branch?

https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160818-gsoc-kconf_symb

Results should help give us an idea of troubling areas if this were to be added
as a new Kconfig feature even for R&D.

  Luis

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Cristina-Gabriela Moraru]

2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
>> Add  CONFIG symbol to kernel modules as a define via -D
>
> Perhaps better worded as:
>
> When modules have a direct Kconfig CONFIG_ symbol associated with
> we want to be able to make it available to the build system when we
> are building the module.
>
> You can then describe you do this with -D.
>
>> compilation parameter. The CONFIG_FOO symbol for each
>> module is determined by the module name, using the
>> associations from Module.ksymb. This file is loaded
>> using the 'include' directive, thus run like a regular
>> makefile. The format of the content of the file is the
>> following:
>>
>> foo_KCONF=CONFIG_FOO
>>
>> creating a set of Makefile variables foo_KCONF with the
>> CONFIG_FOO as values.
>>
>> The Makefile then adds this value in the compilation
>> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
>
> Very nice. What would it mean if it lacks this ?
> This should be explained on the commit log.
>

If we lack it then KBUILD_KCONF="" and in /sys the module has the
kconfig_symbol attribute but with the empty string as content:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

I will add it into the commit log.

>> This patch is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> You can leave this out as its part of the cover letter and
> not relevant to the patch. You can however mention you will
> use this later to make the define more useful in userspace.
>
>> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
>> ---
>>  scripts/Makefile.lib | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
>> index e7df0f5..8ae9b7f 100644
>> --- a/scripts/Makefile.lib
>> +++ b/scripts/Makefile.lib
>> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
>>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
>>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
>>
>> +# Include Module.ksymb which contains the associations of modules' names
>> +# and corresponding CONFIG_* options
>> +include Module.ksymb
>
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

Yes. This file is always generated. In case a 'git pull' happened
between two 'make' commands, the associations should be updated. Ok. I
will add a ifdef.

>
>> +
>>  # These flags are needed for modversions and compiling, so we define them here
>>  # already
>>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
>> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
>> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
>> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
>> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
>
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

I think the fact that modname is unique prevents from having clashes.
KBUILD_KSYMB is found according to modname.
Currently there is no clash because I enforced the Module.ksymb to
have 1-to-1 mapping.
The only potential clash I can imagine right now it having the same
module name but architecture specific CONFIG_* symbol. If there is a
1-to-1 mapping in Module.ksymb there should be no clash in the
makefile.

>
>>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
>> @@ -162,7 +169,7 @@ endif
>>
>>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__c_flags) $(modkern_cflags)                           \
>> -              $(basename_flags) $(modname_flags)
>> +              $(basename_flags) $(modname_flags) $(ksymb_flags)
>>
>>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__a_flags) $(modkern_aflags)
>> --
>
> Other than that looks good!
>
>   Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Cristina-Gabriela Moraru]

2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
>> Add  CONFIG symbol to kernel modules as a define via -D
>
> Perhaps better worded as:
>
> When modules have a direct Kconfig CONFIG_ symbol associated with
> we want to be able to make it available to the build system when we
> are building the module.
>
> You can then describe you do this with -D.
>
>> compilation parameter. The CONFIG_FOO symbol for each
>> module is determined by the module name, using the
>> associations from Module.ksymb. This file is loaded
>> using the 'include' directive, thus run like a regular
>> makefile. The format of the content of the file is the
>> following:
>>
>> foo_KCONF=CONFIG_FOO
>>
>> creating a set of Makefile variables foo_KCONF with the
>> CONFIG_FOO as values.
>>
>> The Makefile then adds this value in the compilation
>> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
>
> Very nice. What would it mean if it lacks this ?
> This should be explained on the commit log.
>

If we lack it then KBUILD_KCONF="" and in /sys the module has the
kconfig_symbol attribute but with the empty string as content:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

I will add it into the commit log.

>> This patch is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> You can leave this out as its part of the cover letter and
> not relevant to the patch. You can however mention you will
> use this later to make the define more useful in userspace.
>
>> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
>> ---
>>  scripts/Makefile.lib | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
>> index e7df0f5..8ae9b7f 100644
>> --- a/scripts/Makefile.lib
>> +++ b/scripts/Makefile.lib
>> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
>>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
>>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
>>
>> +# Include Module.ksymb which contains the associations of modules' names
>> +# and corresponding CONFIG_* options
>> +include Module.ksymb
>
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

Yes. This file is always generated. In case a 'git pull' happened
between two 'make' commands, the associations should be updated. Ok. I
will add a ifdef.

>
>> +
>>  # These flags are needed for modversions and compiling, so we define them here
>>  # already
>>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
>> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
>> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
>> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
>> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
>
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

I think the fact that modname is unique prevents from having clashes.
KBUILD_KSYMB is found according to modname.
Currently there is no clash because I enforced the Module.ksymb to
have 1-to-1 mapping.
The only potential clash I can imagine right now it having the same
module name but architecture specific CONFIG_* symbol. If there is a
1-to-1 mapping in Module.ksymb there should be no clash in the
makefile.

>
>>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
>> @@ -162,7 +169,7 @@ endif
>>
>>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__c_flags) $(modkern_cflags)                           \
>> -              $(basename_flags) $(modname_flags)
>> +              $(basename_flags) $(modname_flags) $(ksymb_flags)
>>
>>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__a_flags) $(modkern_aflags)
>> --
>
> Other than that looks good!
>
>   Luis

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Luis R. Rodriguez]

On Sat, Aug 20, 2016 at 05:11:37PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> > On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> >> Add  CONFIG symbol to kernel modules as a define via -D
> >
> > Perhaps better worded as:
> >
> > When modules have a direct Kconfig CONFIG_ symbol associated with
> > we want to be able to make it available to the build system when we
> > are building the module.
> >
> > You can then describe you do this with -D.
> >
> >> compilation parameter. The CONFIG_FOO symbol for each
> >> module is determined by the module name, using the
> >> associations from Module.ksymb. This file is loaded
> >> using the 'include' directive, thus run like a regular
> >> makefile. The format of the content of the file is the
> >> following:
> >>
> >> foo_KCONF=CONFIG_FOO
> >>
> >> creating a set of Makefile variables foo_KCONF with the
> >> CONFIG_FOO as values.
> >>
> >> The Makefile then adds this value in the compilation
> >> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
> >
> > Very nice. What would it mean if it lacks this ?
> > This should be explained on the commit log.
> >
> 
> If we lack it then KBUILD_KCONF="" and in /sys the module has the
> kconfig_symbol attribute but with the empty string as content:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$
> 
> I will add it into the commit log.

OK -- I do wonder if instead of an empty string leaving the kconfig_symbol
out is better. If its empty then it can be confusing, and so perhaps better
an "unknown" is better. But skipping the attribute then seems best as then
we can focus on just addressing what it *does mean* when we do have a mapping.
This would allow addressing the semantic gap of modules that lack this and
trying to fix those step by step. To fix those we first need to identify
*why* we can't get an attribute pegged to these - document this perhaps on
kernelnewbies.org/KernelProjects/<pick-a-topic-name-for-your-project>
and then your commit log can reference this for a list of known reasons
and pending work.

> >> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> >> index e7df0f5..8ae9b7f 100644
> >> --- a/scripts/Makefile.lib
> >> +++ b/scripts/Makefile.lib
> >> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
> >>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
> >>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
> >>
> >> +# Include Module.ksymb which contains the associations of modules' names
> >> +# and corresponding CONFIG_* options
> >> +include Module.ksymb
> >
> > Is this file always generated? If not prefixing the include call with - would
> > be better. If we are going to make this optional, then an ifdef wrapper would
> > suffice as we know it'd be expected only if the new feature was enabled.
> 
> Yes. This file is always generated. In case a 'git pull' happened
> between two 'make' commands, the associations should be updated. Ok. I
> will add a ifdef.
> 
> >
> >> +
> >>  # These flags are needed for modversions and compiling, so we define them here
> >>  # already
> >>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> >> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> >> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> >> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> >> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> >
> > Are clashes possible with this formula? Can we end up with two results for instance?
> > If not what prevents current konfig logic and namespace from a clash ? If we do
> > not have anything to prevent a clash, what can we do to help make such clash not
> > possible ?
> 
> I think the fact that modname is unique prevents from having clashes.
> KBUILD_KSYMB is found according to modname.
> Currently there is no clash because I enforced the Module.ksymb to
> have 1-to-1 mapping.
> The only potential clash I can imagine right now it having the same
> module name but architecture specific CONFIG_* symbol. If there is a
> 1-to-1 mapping in Module.ksymb there should be no clash in the
> makefile.

OK thanks.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

[Luis R. Rodriguez]

On Sat, Aug 20, 2016 at 05:11:37PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> > On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> >> Add  CONFIG symbol to kernel modules as a define via -D
> >
> > Perhaps better worded as:
> >
> > When modules have a direct Kconfig CONFIG_ symbol associated with
> > we want to be able to make it available to the build system when we
> > are building the module.
> >
> > You can then describe you do this with -D.
> >
> >> compilation parameter. The CONFIG_FOO symbol for each
> >> module is determined by the module name, using the
> >> associations from Module.ksymb. This file is loaded
> >> using the 'include' directive, thus run like a regular
> >> makefile. The format of the content of the file is the
> >> following:
> >>
> >> foo_KCONF=CONFIG_FOO
> >>
> >> creating a set of Makefile variables foo_KCONF with the
> >> CONFIG_FOO as values.
> >>
> >> The Makefile then adds this value in the compilation
> >> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
> >
> > Very nice. What would it mean if it lacks this ?
> > This should be explained on the commit log.
> >
> 
> If we lack it then KBUILD_KCONF="" and in /sys the module has the
> kconfig_symbol attribute but with the empty string as content:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$
> 
> I will add it into the commit log.

OK -- I do wonder if instead of an empty string leaving the kconfig_symbol
out is better. If its empty then it can be confusing, and so perhaps better
an "unknown" is better. But skipping the attribute then seems best as then
we can focus on just addressing what it *does mean* when we do have a mapping.
This would allow addressing the semantic gap of modules that lack this and
trying to fix those step by step. To fix those we first need to identify
*why* we can't get an attribute pegged to these - document this perhaps on
kernelnewbies.org/KernelProjects/<pick-a-topic-name-for-your-project>
and then your commit log can reference this for a list of known reasons
and pending work.

> >> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> >> index e7df0f5..8ae9b7f 100644
> >> --- a/scripts/Makefile.lib
> >> +++ b/scripts/Makefile.lib
> >> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
> >>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
> >>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
> >>
> >> +# Include Module.ksymb which contains the associations of modules' names
> >> +# and corresponding CONFIG_* options
> >> +include Module.ksymb
> >
> > Is this file always generated? If not prefixing the include call with - would
> > be better. If we are going to make this optional, then an ifdef wrapper would
> > suffice as we know it'd be expected only if the new feature was enabled.
> 
> Yes. This file is always generated. In case a 'git pull' happened
> between two 'make' commands, the associations should be updated. Ok. I
> will add a ifdef.
> 
> >
> >> +
> >>  # These flags are needed for modversions and compiling, so we define them here
> >>  # already
> >>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> >> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> >> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> >> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> >> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> >
> > Are clashes possible with this formula? Can we end up with two results for instance?
> > If not what prevents current konfig logic and namespace from a clash ? If we do
> > not have anything to prevent a clash, what can we do to help make such clash not
> > possible ?
> 
> I think the fact that modname is unique prevents from having clashes.
> KBUILD_KSYMB is found according to modname.
> Currently there is no clash because I enforced the Module.ksymb to
> have 1-to-1 mapping.
> The only potential clash I can imagine right now it having the same
> module name but architecture specific CONFIG_* symbol. If there is a
> 1-to-1 mapping in Module.ksymb there should be no clash in the
> makefile.

OK thanks.

  Luis

[RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile

[Cristina Moraru]

Trigger the generation of file scripts/Module.ksymb in
Makefile by calling updated scripts/streamline_config.pl
with corresponding parameter (--genmodulesymb). The file
is generated at each compilation considering that
associations may change after tree updates.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Makefile b/Makefile
index 393b615..286b949 100644
--- a/Makefile
+++ b/Makefile
@@ -219,6 +219,10 @@ VPATH		:= $(srctree)$(if $(KBUILD_EXTMOD),:$(KBUILD_EXTMOD))
 
 export srctree objtree VPATH
 
+KSYMB_GENERATOR := $(objtree)/scripts/kconfig/streamline_config.pl
+ksymb_gen_command = perl $(KSYMB_GENERATOR) --genmoduleksymb $(objtree) $(Kconfig)
+ksymb_update := $(shell objtree=$(objtree) srctree=$(srctree) $(ksymb_gen_command))
+
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
 # line overrides the setting of ARCH below.  If a native build is happening,
-- 
2.7.4

Re: [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:01PM +0200, Cristina Moraru wrote:
> Trigger the generation of file scripts/Module.ksymb in
> Makefile by calling updated scripts/streamline_config.pl
> with corresponding parameter (--genmodulesymb). The file
> is generated at each compilation considering that
> associations may change after tree updates.

I really like this approach as you make the resolver
configurable, this lets others experiment and consider
alternatives.

make KSYMB_GENERATOR=/path/research/my-config-generator-is-better

The fact that you can do this should be documented as part of
the patch commit log.

You'll want to wrap this generation under a CONFIG option so only
kernels that want generate this file.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> 
> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 393b615..286b949 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -219,6 +219,10 @@ VPATH		:= $(srctree)$(if $(KBUILD_EXTMOD),:$(KBUILD_EXTMOD))
>  
>  export srctree objtree VPATH
>  
> +KSYMB_GENERATOR := $(objtree)/scripts/kconfig/streamline_config.pl
> +ksymb_gen_command = perl $(KSYMB_GENERATOR) --genmoduleksymb $(objtree) $(Kconfig)

$(PERL)

> +ksymb_update := $(shell objtree=$(objtree) srctree=$(srctree) $(ksymb_gen_command))
> +
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
>  # line overrides the setting of ARCH below.  If a native build is happening,
> -- 
> 2.7.4

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:01PM +0200, Cristina Moraru wrote:
> Trigger the generation of file scripts/Module.ksymb in
> Makefile by calling updated scripts/streamline_config.pl
> with corresponding parameter (--genmodulesymb). The file
> is generated at each compilation considering that
> associations may change after tree updates.

I really like this approach as you make the resolver
configurable, this lets others experiment and consider
alternatives.

make KSYMB_GENERATOR=/path/research/my-config-generator-is-better

The fact that you can do this should be documented as part of
the patch commit log.

You'll want to wrap this generation under a CONFIG option so only
kernels that want generate this file.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> 
> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 393b615..286b949 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -219,6 +219,10 @@ VPATH		:= $(srctree)$(if $(KBUILD_EXTMOD),:$(KBUILD_EXTMOD))
>  
>  export srctree objtree VPATH
>  
> +KSYMB_GENERATOR := $(objtree)/scripts/kconfig/streamline_config.pl
> +ksymb_gen_command = perl $(KSYMB_GENERATOR) --genmoduleksymb $(objtree) $(Kconfig)

$(PERL)

> +ksymb_update := $(shell objtree=$(objtree) srctree=$(srctree) $(ksymb_gen_command))
> +
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
>  # line overrides the setting of ARCH below.  If a native build is happening,
> -- 
> 2.7.4

  Luis

[RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

[Cristina Moraru]

Update modpost to add in *.mod.c files generated for each
module the setting of module attribute kernel_ksymb to
value given by KBUILD_KSYMB macro.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 scripts/mod/modpost.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 48958d3..a105916 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
 	}
 }
 
+static void add_kconfig_symbol(struct buffer *b, struct module *mod)
+{
+	buf_printf(b, "\n");
+	buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
+}
+
 static void write_if_changed(struct buffer *b, const char *fname)
 {
 	char *tmp;
@@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
 		add_depends(&buf, mod, modules);
 		add_moddevtable(&buf, mod);
 		add_srcversion(&buf, mod);
+		add_kconfig_symbol(&buf, mod);
 
 		sprintf(fname, "%s.mod.c", mod->name);
 		write_if_changed(&buf, fname);
-- 
2.7.4

Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> Update modpost to add in *.mod.c files generated for each
> module the setting of module attribute kernel_ksymb to
> value given by KBUILD_KSYMB macro.

Please review your patches and update subjects to match what other
types of previous patches look like in terms of format, so
for this patch check changes to scripts/mod/modpost.c and use
similar type of patch prefix.

> 
> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/mod/modpost.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> index 48958d3..a105916 100644
> --- a/scripts/mod/modpost.c
> +++ b/scripts/mod/modpost.c
> @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
>  	}
>  }
>  
> +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> +{
> +	buf_printf(b, "\n");
> +	buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");

What if its not available? What happens?

> +}
> +
>  static void write_if_changed(struct buffer *b, const char *fname)
>  {
>  	char *tmp;
> @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
>  		add_depends(&buf, mod, modules);
>  		add_moddevtable(&buf, mod);
>  		add_srcversion(&buf, mod);
> +		add_kconfig_symbol(&buf, mod);
>  
>  		sprintf(fname, "%s.mod.c", mod->name);
>  		write_if_changed(&buf, fname);
> -- 
> 2.7.4
> 
> 

-- 
Luis Rodriguez, SUSE LINUX GmbH
Maxfeldstrasse 5; D-90409 Nuernberg
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> Update modpost to add in *.mod.c files generated for each
> module the setting of module attribute kernel_ksymb to
> value given by KBUILD_KSYMB macro.

Please review your patches and update subjects to match what other
types of previous patches look like in terms of format, so
for this patch check changes to scripts/mod/modpost.c and use
similar type of patch prefix.

> 
> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/mod/modpost.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> index 48958d3..a105916 100644
> --- a/scripts/mod/modpost.c
> +++ b/scripts/mod/modpost.c
> @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
>  	}
>  }
>  
> +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> +{
> +	buf_printf(b, "\n");
> +	buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");

What if its not available? What happens?

> +}
> +
>  static void write_if_changed(struct buffer *b, const char *fname)
>  {
>  	char *tmp;
> @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
>  		add_depends(&buf, mod, modules);
>  		add_moddevtable(&buf, mod);
>  		add_srcversion(&buf, mod);
> +		add_kconfig_symbol(&buf, mod);
>  
>  		sprintf(fname, "%s.mod.c", mod->name);
>  		write_if_changed(&buf, fname);
> -- 
> 2.7.4
> 
> 

-- 
Luis Rodriguez, SUSE LINUX GmbH
Maxfeldstrasse 5; D-90409 Nuernberg

Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

[Cristina-Gabriela Moraru]

2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
>
> On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > Update modpost to add in *.mod.c files generated for each
> > module the setting of module attribute kernel_ksymb to
> > value given by KBUILD_KSYMB macro.
>
> Please review your patches and update subjects to match what other
> types of previous patches look like in terms of format, so
> for this patch check changes to scripts/mod/modpost.c and use
> similar type of patch prefix.
>

OK

>
> >
> > This patch is part of a research project within
> > Google Summer of Code of porting 'make localmodconfig'
> > for backported drivers. The goal is to enable each
> > module to expose in /sys its corresponding CONFIG_* option.
> > The value of this attribute will be dynamically pegged by
> > modpost without requiring extra work from the driver developers.
> > Further, this information will be used by a hardware interogation
> > tool to extract build information about the existing devices.
>
> You can leave this out.
>
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/mod/modpost.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > index 48958d3..a105916 100644
> > --- a/scripts/mod/modpost.c
> > +++ b/scripts/mod/modpost.c
> > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> >       }
> >  }
> >
> > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > +{
> > +     buf_printf(b, "\n");
> > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
>
> What if its not available? What happens?
>

If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
More concrete:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

>
> > +}
> > +
> >  static void write_if_changed(struct buffer *b, const char *fname)
> >  {
> >       char *tmp;
> > @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
> >               add_depends(&buf, mod, modules);
> >               add_moddevtable(&buf, mod);
> >               add_srcversion(&buf, mod);
> > +             add_kconfig_symbol(&buf, mod);
> >
> >               sprintf(fname, "%s.mod.c", mod->name);
> >               write_if_changed(&buf, fname);
> > --
> > 2.7.4
> >
> >
>
> --
> Luis Rodriguez, SUSE LINUX GmbH
> Maxfeldstrasse 5; D-90409 Nuernberg
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

[Cristina-Gabriela Moraru]

2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
>
> On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > Update modpost to add in *.mod.c files generated for each
> > module the setting of module attribute kernel_ksymb to
> > value given by KBUILD_KSYMB macro.
>
> Please review your patches and update subjects to match what other
> types of previous patches look like in terms of format, so
> for this patch check changes to scripts/mod/modpost.c and use
> similar type of patch prefix.
>

OK

>
> >
> > This patch is part of a research project within
> > Google Summer of Code of porting 'make localmodconfig'
> > for backported drivers. The goal is to enable each
> > module to expose in /sys its corresponding CONFIG_* option.
> > The value of this attribute will be dynamically pegged by
> > modpost without requiring extra work from the driver developers.
> > Further, this information will be used by a hardware interogation
> > tool to extract build information about the existing devices.
>
> You can leave this out.
>
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/mod/modpost.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > index 48958d3..a105916 100644
> > --- a/scripts/mod/modpost.c
> > +++ b/scripts/mod/modpost.c
> > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> >       }
> >  }
> >
> > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > +{
> > +     buf_printf(b, "\n");
> > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
>
> What if its not available? What happens?
>

If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
More concrete:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

>
> > +}
> > +
> >  static void write_if_changed(struct buffer *b, const char *fname)
> >  {
> >       char *tmp;
> > @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
> >               add_depends(&buf, mod, modules);
> >               add_moddevtable(&buf, mod);
> >               add_srcversion(&buf, mod);
> > +             add_kconfig_symbol(&buf, mod);
> >
> >               sprintf(fname, "%s.mod.c", mod->name);
> >               write_if_changed(&buf, fname);
> > --
> > 2.7.4
> >
> >
>
> --
> Luis Rodriguez, SUSE LINUX GmbH
> Maxfeldstrasse 5; D-90409 Nuernberg

Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

[Luis R. Rodriguez]

On Sat, Aug 20, 2016 at 05:16:50PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> >
> > On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > > index 48958d3..a105916 100644
> > > --- a/scripts/mod/modpost.c
> > > +++ b/scripts/mod/modpost.c
> > > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> > >       }
> > >  }
> > >
> > > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > > +{
> > > +     buf_printf(b, "\n");
> > > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
> >
> > What if its not available? What happens?
> >
> 
> If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
> More concrete:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$

As I noted in my other e-mail I think its best we just avoid exposing this
then for modules that lack a mapping, and then we make this best effort,
but document the shortcomings very well and super clearly as to why a
backward map is not possible. This then enables us to do work on the build
system to help with this and complete the semantic gap.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

[Luis R. Rodriguez]

On Sat, Aug 20, 2016 at 05:16:50PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> >
> > On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > > index 48958d3..a105916 100644
> > > --- a/scripts/mod/modpost.c
> > > +++ b/scripts/mod/modpost.c
> > > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> > >       }
> > >  }
> > >
> > > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > > +{
> > > +     buf_printf(b, "\n");
> > > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
> >
> > What if its not available? What happens?
> >
> 
> If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
> More concrete:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$

As I noted in my other e-mail I think its best we just avoid exposing this
then for modules that lack a mapping, and then we make this best effort,
but document the shortcomings very well and super clearly as to why a
backward map is not possible. This then enables us to do work on the build
system to help with this and complete the semantic gap.

  Luis

[RFC PATCH 5/5] Add kconf_symb as kernel module attribute

[Cristina Moraru]

Add kconf_symbol attribute to kernel struct module.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 include/linux/module.h | 1 +
 kernel/module.c        | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/include/linux/module.h b/include/linux/module.h
index 3daf2b3..bef5e44 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -353,6 +353,7 @@ struct module {
 	struct module_attribute *modinfo_attrs;
 	const char *version;
 	const char *srcversion;
+	const char *kconfig_symbol;
 	struct kobject *holders_dir;
 
 	/* Exported symbols */
diff --git a/kernel/module.c b/kernel/module.c
index 5f71aa6..bb27300 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -757,6 +757,7 @@ static struct module_attribute modinfo_##field = {                    \
 
 MODINFO_ATTR(version);
 MODINFO_ATTR(srcversion);
+MODINFO_ATTR(kconfig_symbol);
 
 static char last_unloaded_module[MODULE_NAME_LEN+1];
 
@@ -1229,6 +1230,7 @@ static struct module_attribute *modinfo_attrs[] = {
 	&module_uevent,
 	&modinfo_version,
 	&modinfo_srcversion,
+	&modinfo_kconfig_symbol,
 	&modinfo_initstate,
 	&modinfo_coresize,
 	&modinfo_initsize,
-- 
2.7.4

Re: [RFC PATCH 5/5] Add kconf_symb as kernel module attribute

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:03PM +0200, Cristina Moraru wrote:
> Add kconf_symbol attribute to kernel struct module.

Nice, you can describe here what and how this can be
used by userspace, for instance you can mention how
udevadm might use this, etc.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  include/linux/module.h | 1 +
>  kernel/module.c        | 2 ++
>  2 files changed, 3 insertions(+)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3daf2b3..bef5e44 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -353,6 +353,7 @@ struct module {
>  	struct module_attribute *modinfo_attrs;
>  	const char *version;
>  	const char *srcversion;
> +	const char *kconfig_symbol;
>  	struct kobject *holders_dir;
>  
>  	/* Exported symbols */
> diff --git a/kernel/module.c b/kernel/module.c
> index 5f71aa6..bb27300 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -757,6 +757,7 @@ static struct module_attribute modinfo_##field = {                    \
>  
>  MODINFO_ATTR(version);
>  MODINFO_ATTR(srcversion);
> +MODINFO_ATTR(kconfig_symbol);
>  
>  static char last_unloaded_module[MODULE_NAME_LEN+1];
>  
> @@ -1229,6 +1230,7 @@ static struct module_attribute *modinfo_attrs[] = {
>  	&module_uevent,
>  	&modinfo_version,
>  	&modinfo_srcversion,
> +	&modinfo_kconfig_symbol,
>  	&modinfo_initstate,
>  	&modinfo_coresize,
>  	&modinfo_initsize,
> -- 

Do we #ifdef over other features that are optional?

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 5/5] Add kconf_symb as kernel module attribute

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:27:03PM +0200, Cristina Moraru wrote:
> Add kconf_symbol attribute to kernel struct module.

Nice, you can describe here what and how this can be
used by userspace, for instance you can mention how
udevadm might use this, etc.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  include/linux/module.h | 1 +
>  kernel/module.c        | 2 ++
>  2 files changed, 3 insertions(+)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3daf2b3..bef5e44 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -353,6 +353,7 @@ struct module {
>  	struct module_attribute *modinfo_attrs;
>  	const char *version;
>  	const char *srcversion;
> +	const char *kconfig_symbol;
>  	struct kobject *holders_dir;
>  
>  	/* Exported symbols */
> diff --git a/kernel/module.c b/kernel/module.c
> index 5f71aa6..bb27300 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -757,6 +757,7 @@ static struct module_attribute modinfo_##field = {                    \
>  
>  MODINFO_ATTR(version);
>  MODINFO_ATTR(srcversion);
> +MODINFO_ATTR(kconfig_symbol);
>  
>  static char last_unloaded_module[MODULE_NAME_LEN+1];
>  
> @@ -1229,6 +1230,7 @@ static struct module_attribute *modinfo_attrs[] = {
>  	&module_uevent,
>  	&modinfo_version,
>  	&modinfo_srcversion,
> +	&modinfo_kconfig_symbol,
>  	&modinfo_initstate,
>  	&modinfo_coresize,
>  	&modinfo_initsize,
> -- 

Do we #ifdef over other features that are optional?

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:

> This patchset implements dynamic pegging of kconfig symbol
> into driver modinfo section

First a little bit of motivation here helps, so let me try to
help fill in some gaps. This may help explain what you have
been working on a bit more.

First, for those that were not Cc'd but curious about this work
so far, you can read the patches here:

Original cover letter:

https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com

https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5

There are a few situations in which you may want to extract a
subset of Kconfig CONFIG_* symbols for a system. At least when
only considering modules:

 a) When optimizing build requirements for a kernel for a system.
    That is you boot into a distro kernel and then want to build
    a slim kernel only with sensible kernel configuration options.

 b) When you are on a distribution kernel but the distribution
    kernel provided lacks hardware support for your device, you
    may either want to upgrade the full kernel in which case you
    want to do a) or -- you may want to just a backports release
    which provides just the modules you need, you'd use it on top
    of the distribution kernel.

Are there other *uses* outside of the two above for having a direct
deterministic mapping between module and a kconfig symbol ? What
if we could also do this for built-in code as well ? kconfig-sat [0] is
work that is still being ironed out, but if that was in place and we
handed it a set of required options we want, it should in theory help
resolve the dependencies to give us a functional .config for a kernel.
This of course is a long term possible goal just used as an example
-- but are there other uses?

Another possible gain here is if addressing a deterministic mapping might
help with some of the limitations observed in practice over years with
kconfig and semantics for keconfig. I go into some of this further below.

There are not many options currently available to optimize kernel builds this
way for modules or choose only what kernel kconfig options you need.  One of
the best known efforts is the upstream 'make localmodconfig' added by Steven
years ago to help with a). This has its own set of limitations though. For
backports we also need to address possible kconfig symbol name changes, and the
fact that you may boot a box without a driver loaded and as such not be sure
what new kernel kconfig symbol you may need.

There are shared issues with both goals, addressing shared issues
can ultimately help both. One of the shared known issues is the
lack of a deterministic resolution of a module to a respective
kconfig symbol. Even answering if this is possible or should be
possible requires some work.

The work Cristina has done here for this year's Google Summer of
Code is try to help start looking into addressing the question if
this is possible and then start looking into how we could annotate
such a thing, and later how we'd use it.

[0] https://kernelnewbies.org/KernelProjects/kconfig-sat

> * updates streamline_config.pl to generate the auxiliary file
> scripts/mod/Module.ksymb containing associations of driver file
> names and corresponding kconfig symbols CONFIG_*

Here you make use of existing heuristics -- note however that
heuristics are needed as we have no direct mapping available
nor do we have any rules over how to make such a direct mapping
possible if we wanted this mapping to be deterministic.

> * updates Makefiles to trigger streamline_config.pl call for
> Module.ksymb generation and pass the determined CONFIG_* symbol
> as compilation parameter -D in KBUILD_KSYMB variable

Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
to the build system a config symbol associated with a module, its the right
approach, however re-using streamline_config.pl seems questionable here, and it
seems an optimization might be / should be possible. Likewise we'd need to also
still look into and address *why* some modules don't end up with a respective
kconfig symbol -- and if we wanted to do something about this. Given the stated
goals more importantly would be what the costs are to have a deterministic
mapping.

> * adds kconfig_symb as module attribute

This seems lightweight and follows the logic of the other attributes.
If its too much, this can be a configurable feature.

> * updates modpost to set KBUILD_KSYMB macro as value for
> kbuild_symb attribute

Neat.

> Note: the content of the file Module.ksymb is generated for
> all modules that have only one associate CONFIG option. All
> others are considered to be components linked at the final
> modules but not final modules itselves.

To be clear, you only peg the CONFIG_ option if you are
100% certain of having a direct mapping, is that right ?

> The result of this patchset is the following. All modules from
> /sys expose the correct CONFIG_* symbol in the module attribute
> kconfig_symb with some exceptions. For a total number of 58
> modules, 

What kernel configuration did you use ? What architecture ?
If you use allmodconfig, what do the numbers look like ?

> 4 of them do not:
> 
> snd_seq_midi_event

Takashi, any reason for this to not have a Kconfig symbol ?
If we don't want one, is there perhaps any annotation we
could use to help make a deterministic association if a
respective kconfig symbol is needed ?

> mptscsih

Same thing -- for both I suppose we can infer that these
are just helpers, and can / should be enabled if any of
the devices are found to be present / required.

I can imagine that in certain situations a helper like
the above might really just be optional rather than required,
but I'd hope such an optional nature should be reflected via
Kconfig, the default setting being the recommended option to
assume as safe then, unless input is provided to help fill
in the gap and disable / force enable certain options.

> libahci

In this case libahci is spinkled over a lot of lines on the Makefile,
its a module though, and there is no direct association of dependency
relationship other than the Makefile association. Using a kconfig
symbol would replace the Makefile clutter but make the assocation
and requirement of the dependency explicit through Kconfig. Is
that desirable or should we just make the assumption that in these
cases we could assume one Kconfig symbol that adds a module implicitly
is equivalent as if the module had a kconfig symbol and we had done
"select foo" for each ? Note though that select is heavy handed [1] --
and its over use can lead to issues. For instance its known that
select does not check for dependencies, its a heavy handed way of
enabling options needed by architectures. So using select here may
not be desirable. Using "depends on" here would work but it can hide
from the menu drivers until you have enabled its dependency.

[1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e

> mptbase

Same situation as  mptscsih.

> After a short research:
> 
> For mptscsih - ./drivers/message/fusion/Makefile
> obj-$(CONFIG_FUSION_SPI)	+= mptbase.o mptscsih.o mptspi.o
> obj-$(CONFIG_FUSION_FC)		+= mptbase.o mptscsih.o mptfc.o
> obj-$(CONFIG_FUSION_SAS)	+= mptbase.o mptscsih.o mptsas.o
> As appears in the Makefile mptscsi is part of more config
> options so it's excluded because it's not considered a module
> by itself but, somehow gets compiled as module. Same for the
> mptbase. I still have to understand better the Kbuild logic so
> I cannot state that this is loose practice.

Another reason for this loose use could also be that using "select"
in Kconfig is heavy handed and can lead to recursive issues, refer
to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
dependency limitation / resolution") for more information and
a clear example of the issue. There is also a list of commits
with possible resolutions to issues, so that can be inspected further
to try to understand if perhaps some of the lack of kconfig symbols
and respective deterministic mapping from module to kconfig is due
to prior issues with select. I doubt it given that otherwise we'd
have at least symbols for these -- but worth taking into consideration
*if* we do then want to add symbols for them, we do not want to lead
to recursive issues if we add these new symbols.

If we add a kconfig symbol for modules lacking them, what is a
strategical way to do so in such a way we avoid known past
issues but also paves the way for the future ?

> For libahci there are multiple CONFIG_ symbols for different
> architectures.
> 
> Although past discussion recommended not to use streamline_config,
> this patchset still relies on it, for the sake of the proof of
> concept. 

OK.

> Although the result is close to the target it still does
> not provide complete correctness. However it can be replaced by
> creating another script

Or C code.

> which tries to determine better the
> module <-> CONFIG associations and output them in auxiliary file
> Module.ksymb. Maybe this way we could also determine all CONFIGs
> for a particular driver, not only the exact one that enables it.
> 
> The auxiliary file is necessary because the Makefile itself does
> not have a mapping. 

Clarification: at least not an easy O(1) quick mapping.

> The makefile includes the config file with
> directive include which creates a series of internal variables
> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
> other Makefiles, lines like 
> 
> obj-$(CONFIG_FOO) = foo.o 
> 
> are resolved in place to obj-y, obj-m etc according to 'make'
> logic and the association is lost.
> 
> Finally, what does this patchset provide is an infrastructure
> to dinamically peg CONFIG_* options to associate drivers using
> the mapping from Module.ksymb file. Generation of Module.ksymb
> can be replaced but keeping the same format permit the usage of
> the other patches.

OK this makes sense, so as it stands it seems we need a better
way to generate the Module.ksymb and then also address those
modules which lack a current deterministic mapping, and address
a resolution with the community on that.

> This patchset is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

Good stuff!

  Luis

> Cristina Moraru (5):
>   Add generation of Module.symb in streamline_config
>   Add CONFIG symbol to module as compilation parameter
>   Trigger Module.ksymb generation in Makefile
>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>   Add kconf_symb as kernel module attribute
> 
>  Makefile                             |  4 ++++
>  include/linux/module.h               |  1 +
>  kernel/module.c                      |  2 ++
>  scripts/Makefile.lib                 |  9 ++++++++-
>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>  scripts/mod/modpost.c                |  7 +++++++
>  6 files changed, 51 insertions(+), 2 deletions(-)
> 
> -- 
> 2.7.4
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:

> This patchset implements dynamic pegging of kconfig symbol
> into driver modinfo section

First a little bit of motivation here helps, so let me try to
help fill in some gaps. This may help explain what you have
been working on a bit more.

First, for those that were not Cc'd but curious about this work
so far, you can read the patches here:

Original cover letter:

https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com

https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5

There are a few situations in which you may want to extract a
subset of Kconfig CONFIG_* symbols for a system. At least when
only considering modules:

 a) When optimizing build requirements for a kernel for a system.
    That is you boot into a distro kernel and then want to build
    a slim kernel only with sensible kernel configuration options.

 b) When you are on a distribution kernel but the distribution
    kernel provided lacks hardware support for your device, you
    may either want to upgrade the full kernel in which case you
    want to do a) or -- you may want to just a backports release
    which provides just the modules you need, you'd use it on top
    of the distribution kernel.

Are there other *uses* outside of the two above for having a direct
deterministic mapping between module and a kconfig symbol ? What
if we could also do this for built-in code as well ? kconfig-sat [0] is
work that is still being ironed out, but if that was in place and we
handed it a set of required options we want, it should in theory help
resolve the dependencies to give us a functional .config for a kernel.
This of course is a long term possible goal just used as an example
-- but are there other uses?

Another possible gain here is if addressing a deterministic mapping might
help with some of the limitations observed in practice over years with
kconfig and semantics for keconfig. I go into some of this further below.

There are not many options currently available to optimize kernel builds this
way for modules or choose only what kernel kconfig options you need.  One of
the best known efforts is the upstream 'make localmodconfig' added by Steven
years ago to help with a). This has its own set of limitations though. For
backports we also need to address possible kconfig symbol name changes, and the
fact that you may boot a box without a driver loaded and as such not be sure
what new kernel kconfig symbol you may need.

There are shared issues with both goals, addressing shared issues
can ultimately help both. One of the shared known issues is the
lack of a deterministic resolution of a module to a respective
kconfig symbol. Even answering if this is possible or should be
possible requires some work.

The work Cristina has done here for this year's Google Summer of
Code is try to help start looking into addressing the question if
this is possible and then start looking into how we could annotate
such a thing, and later how we'd use it.

[0] https://kernelnewbies.org/KernelProjects/kconfig-sat

> * updates streamline_config.pl to generate the auxiliary file
> scripts/mod/Module.ksymb containing associations of driver file
> names and corresponding kconfig symbols CONFIG_*

Here you make use of existing heuristics -- note however that
heuristics are needed as we have no direct mapping available
nor do we have any rules over how to make such a direct mapping
possible if we wanted this mapping to be deterministic.

> * updates Makefiles to trigger streamline_config.pl call for
> Module.ksymb generation and pass the determined CONFIG_* symbol
> as compilation parameter -D in KBUILD_KSYMB variable

Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
to the build system a config symbol associated with a module, its the right
approach, however re-using streamline_config.pl seems questionable here, and it
seems an optimization might be / should be possible. Likewise we'd need to also
still look into and address *why* some modules don't end up with a respective
kconfig symbol -- and if we wanted to do something about this. Given the stated
goals more importantly would be what the costs are to have a deterministic
mapping.

> * adds kconfig_symb as module attribute

This seems lightweight and follows the logic of the other attributes.
If its too much, this can be a configurable feature.

> * updates modpost to set KBUILD_KSYMB macro as value for
> kbuild_symb attribute

Neat.

> Note: the content of the file Module.ksymb is generated for
> all modules that have only one associate CONFIG option. All
> others are considered to be components linked at the final
> modules but not final modules itselves.

To be clear, you only peg the CONFIG_ option if you are
100% certain of having a direct mapping, is that right ?

> The result of this patchset is the following. All modules from
> /sys expose the correct CONFIG_* symbol in the module attribute
> kconfig_symb with some exceptions. For a total number of 58
> modules, 

What kernel configuration did you use ? What architecture ?
If you use allmodconfig, what do the numbers look like ?

> 4 of them do not:
> 
> snd_seq_midi_event

Takashi, any reason for this to not have a Kconfig symbol ?
If we don't want one, is there perhaps any annotation we
could use to help make a deterministic association if a
respective kconfig symbol is needed ?

> mptscsih

Same thing -- for both I suppose we can infer that these
are just helpers, and can / should be enabled if any of
the devices are found to be present / required.

I can imagine that in certain situations a helper like
the above might really just be optional rather than required,
but I'd hope such an optional nature should be reflected via
Kconfig, the default setting being the recommended option to
assume as safe then, unless input is provided to help fill
in the gap and disable / force enable certain options.

> libahci

In this case libahci is spinkled over a lot of lines on the Makefile,
its a module though, and there is no direct association of dependency
relationship other than the Makefile association. Using a kconfig
symbol would replace the Makefile clutter but make the assocation
and requirement of the dependency explicit through Kconfig. Is
that desirable or should we just make the assumption that in these
cases we could assume one Kconfig symbol that adds a module implicitly
is equivalent as if the module had a kconfig symbol and we had done
"select foo" for each ? Note though that select is heavy handed [1] --
and its over use can lead to issues. For instance its known that
select does not check for dependencies, its a heavy handed way of
enabling options needed by architectures. So using select here may
not be desirable. Using "depends on" here would work but it can hide
from the menu drivers until you have enabled its dependency.

[1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e

> mptbase

Same situation as  mptscsih.

> After a short research:
> 
> For mptscsih - ./drivers/message/fusion/Makefile
> obj-$(CONFIG_FUSION_SPI)	+= mptbase.o mptscsih.o mptspi.o
> obj-$(CONFIG_FUSION_FC)		+= mptbase.o mptscsih.o mptfc.o
> obj-$(CONFIG_FUSION_SAS)	+= mptbase.o mptscsih.o mptsas.o
> As appears in the Makefile mptscsi is part of more config
> options so it's excluded because it's not considered a module
> by itself but, somehow gets compiled as module. Same for the
> mptbase. I still have to understand better the Kbuild logic so
> I cannot state that this is loose practice.

Another reason for this loose use could also be that using "select"
in Kconfig is heavy handed and can lead to recursive issues, refer
to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
dependency limitation / resolution") for more information and
a clear example of the issue. There is also a list of commits
with possible resolutions to issues, so that can be inspected further
to try to understand if perhaps some of the lack of kconfig symbols
and respective deterministic mapping from module to kconfig is due
to prior issues with select. I doubt it given that otherwise we'd
have at least symbols for these -- but worth taking into consideration
*if* we do then want to add symbols for them, we do not want to lead
to recursive issues if we add these new symbols.

If we add a kconfig symbol for modules lacking them, what is a
strategical way to do so in such a way we avoid known past
issues but also paves the way for the future ?

> For libahci there are multiple CONFIG_ symbols for different
> architectures.
> 
> Although past discussion recommended not to use streamline_config,
> this patchset still relies on it, for the sake of the proof of
> concept. 

OK.

> Although the result is close to the target it still does
> not provide complete correctness. However it can be replaced by
> creating another script

Or C code.

> which tries to determine better the
> module <-> CONFIG associations and output them in auxiliary file
> Module.ksymb. Maybe this way we could also determine all CONFIGs
> for a particular driver, not only the exact one that enables it.
> 
> The auxiliary file is necessary because the Makefile itself does
> not have a mapping. 

Clarification: at least not an easy O(1) quick mapping.

> The makefile includes the config file with
> directive include which creates a series of internal variables
> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
> other Makefiles, lines like 
> 
> obj-$(CONFIG_FOO) = foo.o 
> 
> are resolved in place to obj-y, obj-m etc according to 'make'
> logic and the association is lost.
> 
> Finally, what does this patchset provide is an infrastructure
> to dinamically peg CONFIG_* options to associate drivers using
> the mapping from Module.ksymb file. Generation of Module.ksymb
> can be replaced but keeping the same format permit the usage of
> the other patches.

OK this makes sense, so as it stands it seems we need a better
way to generate the Module.ksymb and then also address those
modules which lack a current deterministic mapping, and address
a resolution with the community on that.

> This patchset is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

Good stuff!

  Luis

> Cristina Moraru (5):
>   Add generation of Module.symb in streamline_config
>   Add CONFIG symbol to module as compilation parameter
>   Trigger Module.ksymb generation in Makefile
>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>   Add kconf_symb as kernel module attribute
> 
>  Makefile                             |  4 ++++
>  include/linux/module.h               |  1 +
>  kernel/module.c                      |  2 ++
>  scripts/Makefile.lib                 |  9 ++++++++-
>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>  scripts/mod/modpost.c                |  7 +++++++
>  6 files changed, 51 insertions(+), 2 deletions(-)
> 
> -- 
> 2.7.4
> 
> 

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Michal Marek]

On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> 
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
> 
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
> 
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
> 
> Original cover letter:
> 
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> 
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> 
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
> 
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
> 
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.

c) Having the mapping in sysfs would allow to simplify
streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
did not depend on streamline_config.pl :). One idea would be to generate
the Module.ksymb in a similar way we generate the modules.builtin file:
Generate an alternate include/config/*.conf with all CONFIG_FOO=m
replaced with

CONFIG_FOO=m-CONFIG_FOO

and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
create the mapping. This would also properly cover cases where we build
the $(obj-m) list from another list. It would certainly create other
corner cases, but it's worth trying IMO.

Another thing is that we do not necessarily need to record this
information in .modinfo, but we can generate a list in
/lib/modules/`uname -r`/ for consumption. It could also include drivers
that are builtin in the current configuration.

Michal
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Michal Marek]

On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> 
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
> 
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
> 
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
> 
> Original cover letter:
> 
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> 
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> 
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
> 
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
> 
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.

c) Having the mapping in sysfs would allow to simplify
streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
did not depend on streamline_config.pl :). One idea would be to generate
the Module.ksymb in a similar way we generate the modules.builtin file:
Generate an alternate include/config/*.conf with all CONFIG_FOO=m
replaced with

CONFIG_FOO=m-CONFIG_FOO

and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
create the mapping. This would also properly cover cases where we build
the $(obj-m) list from another list. It would certainly create other
corner cases, but it's worth trying IMO.

Another thing is that we do not necessarily need to record this
information in .modinfo, but we can generate a list in
/lib/modules/`uname -r`/ for consumption. It could also include drivers
that are builtin in the current configuration.

Michal

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Cristina-Gabriela Moraru]

2016-08-19 11:07 GMT+02:00 Michal Marek <mmarek@suse.com>:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>
>>> This patchset implements dynamic pegging of kconfig symbol
>>> into driver modinfo section
>>
>> First a little bit of motivation here helps, so let me try to
>> help fill in some gaps. This may help explain what you have
>> been working on a bit more.
>>
>> First, for those that were not Cc'd but curious about this work
>> so far, you can read the patches here:
>>
>> Original cover letter:
>>
>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>
>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>
>> There are a few situations in which you may want to extract a
>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> only considering modules:
>>
>>  a) When optimizing build requirements for a kernel for a system.
>>     That is you boot into a distro kernel and then want to build
>>     a slim kernel only with sensible kernel configuration options.
>>
>>  b) When you are on a distribution kernel but the distribution
>>     kernel provided lacks hardware support for your device, you
>>     may either want to upgrade the full kernel in which case you
>>     want to do a) or -- you may want to just a backports release
>>     which provides just the modules you need, you'd use it on top
>>     of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
> did not depend on streamline_config.pl :). One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
>
> CONFIG_FOO=m-CONFIG_FOO
>
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.
>

That's interesting.

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.
>
> Michal

Thank you,

Cristina
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Cristina-Gabriela Moraru]

2016-08-19 11:07 GMT+02:00 Michal Marek <mmarek@suse.com>:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>
>>> This patchset implements dynamic pegging of kconfig symbol
>>> into driver modinfo section
>>
>> First a little bit of motivation here helps, so let me try to
>> help fill in some gaps. This may help explain what you have
>> been working on a bit more.
>>
>> First, for those that were not Cc'd but curious about this work
>> so far, you can read the patches here:
>>
>> Original cover letter:
>>
>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>
>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>
>> There are a few situations in which you may want to extract a
>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> only considering modules:
>>
>>  a) When optimizing build requirements for a kernel for a system.
>>     That is you boot into a distro kernel and then want to build
>>     a slim kernel only with sensible kernel configuration options.
>>
>>  b) When you are on a distribution kernel but the distribution
>>     kernel provided lacks hardware support for your device, you
>>     may either want to upgrade the full kernel in which case you
>>     want to do a) or -- you may want to just a backports release
>>     which provides just the modules you need, you'd use it on top
>>     of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
> did not depend on streamline_config.pl :). One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
>
> CONFIG_FOO=m-CONFIG_FOO
>
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.
>

That's interesting.

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.
>
> Michal

Thank you,

Cristina

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> > 
> >> This patchset implements dynamic pegging of kconfig symbol
> >> into driver modinfo section
> > 
> > First a little bit of motivation here helps, so let me try to
> > help fill in some gaps. This may help explain what you have
> > been working on a bit more.
> > 
> > First, for those that were not Cc'd but curious about this work
> > so far, you can read the patches here:
> > 
> > Original cover letter:
> > 
> > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> > 
> > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> > 
> > There are a few situations in which you may want to extract a
> > subset of Kconfig CONFIG_* symbols for a system. At least when
> > only considering modules:
> > 
> >  a) When optimizing build requirements for a kernel for a system.
> >     That is you boot into a distro kernel and then want to build
> >     a slim kernel only with sensible kernel configuration options.
> > 
> >  b) When you are on a distribution kernel but the distribution
> >     kernel provided lacks hardware support for your device, you
> >     may either want to upgrade the full kernel in which case you
> >     want to do a) or -- you may want to just a backports release
> >     which provides just the modules you need, you'd use it on top
> >     of the distribution kernel.
> 
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl.

Indeed, this was actually a side possible goal here :) only it seems
we totally forgot about it while addressing the work.

> Only if the patch did not depend on streamline_config.pl :)

Indeed, lets get rid of that dependency.

> One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
> 
> CONFIG_FOO=m-CONFIG_FOO
> 
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.

Neat, would that cover lib-m then as well or other targets that would generate 
modules ?

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.

Well so one future use would be for instance a 'make udevconfig' which
in theory should be able then to scrape all possible related needed
device CONFIG_'s and then give you an optimal kernel configuration.
It'd be nice to be able to extract this from the kernel and modules
without needing build sources in /lib/modules/`uname -r`/.

Thoughts?

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> > 
> >> This patchset implements dynamic pegging of kconfig symbol
> >> into driver modinfo section
> > 
> > First a little bit of motivation here helps, so let me try to
> > help fill in some gaps. This may help explain what you have
> > been working on a bit more.
> > 
> > First, for those that were not Cc'd but curious about this work
> > so far, you can read the patches here:
> > 
> > Original cover letter:
> > 
> > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> > 
> > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> > 
> > There are a few situations in which you may want to extract a
> > subset of Kconfig CONFIG_* symbols for a system. At least when
> > only considering modules:
> > 
> >  a) When optimizing build requirements for a kernel for a system.
> >     That is you boot into a distro kernel and then want to build
> >     a slim kernel only with sensible kernel configuration options.
> > 
> >  b) When you are on a distribution kernel but the distribution
> >     kernel provided lacks hardware support for your device, you
> >     may either want to upgrade the full kernel in which case you
> >     want to do a) or -- you may want to just a backports release
> >     which provides just the modules you need, you'd use it on top
> >     of the distribution kernel.
> 
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl.

Indeed, this was actually a side possible goal here :) only it seems
we totally forgot about it while addressing the work.

> Only if the patch did not depend on streamline_config.pl :)

Indeed, lets get rid of that dependency.

> One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
> 
> CONFIG_FOO=m-CONFIG_FOO
> 
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.

Neat, would that cover lib-m then as well or other targets that would generate 
modules ?

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.

Well so one future use would be for instance a 'make udevconfig' which
in theory should be able then to scrape all possible related needed
device CONFIG_'s and then give you an optimal kernel configuration.
It'd be nice to be able to extract this from the kernel and modules
without needing build sources in /lib/modules/`uname -r`/.

Thoughts?

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Michal Marek]

On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>>
>>>> This patchset implements dynamic pegging of kconfig symbol
>>>> into driver modinfo section
>>>
>>> First a little bit of motivation here helps, so let me try to
>>> help fill in some gaps. This may help explain what you have
>>> been working on a bit more.
>>>
>>> First, for those that were not Cc'd but curious about this work
>>> so far, you can read the patches here:
>>>
>>> Original cover letter:
>>>
>>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>>
>>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>>
>>> There are a few situations in which you may want to extract a
>>> subset of Kconfig CONFIG_* symbols for a system. At least when
>>> only considering modules:
>>>
>>>  a) When optimizing build requirements for a kernel for a system.
>>>     That is you boot into a distro kernel and then want to build
>>>     a slim kernel only with sensible kernel configuration options.
>>>
>>>  b) When you are on a distribution kernel but the distribution
>>>     kernel provided lacks hardware support for your device, you
>>>     may either want to upgrade the full kernel in which case you
>>>     want to do a) or -- you may want to just a backports release
>>>     which provides just the modules you need, you'd use it on top
>>>     of the distribution kernel.
>>
>> c) Having the mapping in sysfs would allow to simplify
>> streamline_config.pl avoid parsing Makefiles in perl.
> 
> Indeed, this was actually a side possible goal here :) only it seems
> we totally forgot about it while addressing the work.
> 
>> Only if the patch did not depend on streamline_config.pl :)
> 
> Indeed, lets get rid of that dependency.
> 
>> One idea would be to generate
>> the Module.ksymb in a similar way we generate the modules.builtin file:
>> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> replaced with
>>
>> CONFIG_FOO=m-CONFIG_FOO
>>
>> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
                                       should be obj-m-CONFIG_%

>> create the mapping. This would also properly cover cases where we build
>> the $(obj-m) list from another list. It would certainly create other
>> corner cases, but it's worth trying IMO.
> 
> Neat, would that cover lib-m then as well or other targets that would generate 
> modules ?

This would have to be handled as well by the patch. Obviously it's not
going to be as simple as described in a single paragraph of text :).


>> Another thing is that we do not necessarily need to record this
>> information in .modinfo, but we can generate a list in
>> /lib/modules/`uname -r`/ for consumption. It could also include drivers
>> that are builtin in the current configuration.
> 
> Well so one future use would be for instance a 'make udevconfig' which
> in theory should be able then to scrape all possible related needed
> device CONFIG_'s and then give you an optimal kernel configuration.
> It'd be nice to be able to extract this from the kernel and modules
> without needing build sources in /lib/modules/`uname -r`/.

I meant that the file can be installed by modules_install. Like you have
/lib/modules/`uname -r`/modules.builtin with recent kernels and recent
distro packages. But really, it's a minor issue how the information is
presented to userspace.

Michal
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Michal Marek]

On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>>
>>>> This patchset implements dynamic pegging of kconfig symbol
>>>> into driver modinfo section
>>>
>>> First a little bit of motivation here helps, so let me try to
>>> help fill in some gaps. This may help explain what you have
>>> been working on a bit more.
>>>
>>> First, for those that were not Cc'd but curious about this work
>>> so far, you can read the patches here:
>>>
>>> Original cover letter:
>>>
>>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>>
>>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>>
>>> There are a few situations in which you may want to extract a
>>> subset of Kconfig CONFIG_* symbols for a system. At least when
>>> only considering modules:
>>>
>>>  a) When optimizing build requirements for a kernel for a system.
>>>     That is you boot into a distro kernel and then want to build
>>>     a slim kernel only with sensible kernel configuration options.
>>>
>>>  b) When you are on a distribution kernel but the distribution
>>>     kernel provided lacks hardware support for your device, you
>>>     may either want to upgrade the full kernel in which case you
>>>     want to do a) or -- you may want to just a backports release
>>>     which provides just the modules you need, you'd use it on top
>>>     of the distribution kernel.
>>
>> c) Having the mapping in sysfs would allow to simplify
>> streamline_config.pl avoid parsing Makefiles in perl.
> 
> Indeed, this was actually a side possible goal here :) only it seems
> we totally forgot about it while addressing the work.
> 
>> Only if the patch did not depend on streamline_config.pl :)
> 
> Indeed, lets get rid of that dependency.
> 
>> One idea would be to generate
>> the Module.ksymb in a similar way we generate the modules.builtin file:
>> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> replaced with
>>
>> CONFIG_FOO=m-CONFIG_FOO
>>
>> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
                                       should be obj-m-CONFIG_%

>> create the mapping. This would also properly cover cases where we build
>> the $(obj-m) list from another list. It would certainly create other
>> corner cases, but it's worth trying IMO.
> 
> Neat, would that cover lib-m then as well or other targets that would generate 
> modules ?

This would have to be handled as well by the patch. Obviously it's not
going to be as simple as described in a single paragraph of text :).


>> Another thing is that we do not necessarily need to record this
>> information in .modinfo, but we can generate a list in
>> /lib/modules/`uname -r`/ for consumption. It could also include drivers
>> that are builtin in the current configuration.
> 
> Well so one future use would be for instance a 'make udevconfig' which
> in theory should be able then to scrape all possible related needed
> device CONFIG_'s and then give you an optimal kernel configuration.
> It'd be nice to be able to extract this from the kernel and modules
> without needing build sources in /lib/modules/`uname -r`/.

I meant that the file can be installed by modules_install. Like you have
/lib/modules/`uname -r`/modules.builtin with recent kernels and recent
distro packages. But really, it's a minor issue how the information is
presented to userspace.

Michal

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> >>>
> >>>> This patchset implements dynamic pegging of kconfig symbol
> >>>> into driver modinfo section
> >>>
> >>> First a little bit of motivation here helps, so let me try to
> >>> help fill in some gaps. This may help explain what you have
> >>> been working on a bit more.
> >>>
> >>> First, for those that were not Cc'd but curious about this work
> >>> so far, you can read the patches here:
> >>>
> >>> Original cover letter:
> >>>
> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> >>>
> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> >>>
> >>> There are a few situations in which you may want to extract a
> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
> >>> only considering modules:
> >>>
> >>>  a) When optimizing build requirements for a kernel for a system.
> >>>     That is you boot into a distro kernel and then want to build
> >>>     a slim kernel only with sensible kernel configuration options.
> >>>
> >>>  b) When you are on a distribution kernel but the distribution
> >>>     kernel provided lacks hardware support for your device, you
> >>>     may either want to upgrade the full kernel in which case you
> >>>     want to do a) or -- you may want to just a backports release
> >>>     which provides just the modules you need, you'd use it on top
> >>>     of the distribution kernel.
> >>
> >> c) Having the mapping in sysfs would allow to simplify
> >> streamline_config.pl avoid parsing Makefiles in perl.
> > 
> > Indeed, this was actually a side possible goal here :) only it seems
> > we totally forgot about it while addressing the work.
> > 
> >> Only if the patch did not depend on streamline_config.pl :)
> > 
> > Indeed, lets get rid of that dependency.
> > 
> >> One idea would be to generate
> >> the Module.ksymb in a similar way we generate the modules.builtin file:
> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> >> replaced with
> >>
> >> CONFIG_FOO=m-CONFIG_FOO
> >>
> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
>                                        should be obj-m-CONFIG_%
> 
> >> create the mapping. This would also properly cover cases where we build
> >> the $(obj-m) list from another list. It would certainly create other
> >> corner cases, but it's worth trying IMO.
> > 
> > Neat, would that cover lib-m then as well or other targets that would generate 
> > modules ?
> 
> This would have to be handled as well by the patch. Obviously it's not
> going to be as simple as described in a single paragraph of text :).

Ah OK, it was not clear to me if your suggestion captured all modules already.
There may still be a way. We just have to look harder. Then what precise
target was used would be irrelevant.

> >> Another thing is that we do not necessarily need to record this
> >> information in .modinfo, but we can generate a list in
> >> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> >> that are builtin in the current configuration.
> > 
> > Well so one future use would be for instance a 'make udevconfig' which
> > in theory should be able then to scrape all possible related needed
> > device CONFIG_'s and then give you an optimal kernel configuration.
> > It'd be nice to be able to extract this from the kernel and modules
> > without needing build sources in /lib/modules/`uname -r`/.
> 
> I meant that the file can be installed by modules_install. Like you have
> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
> distro packages.

Got it.

> But really, it's a minor issue how the information is
> presented to userspace.

To us perhaps, but to userspace and users this is a big deal. It'd be nice
to be able to construct a kernel config only with what CONFIG's your hardware
needs, doing this should mean trying to expose enough information in a way
userspace is easily able to handle. The reason to go with the mod section stuff
was this was already being picked up by udevadm for its own hardware db info,
so this seemed like a logical evolution, and it would not require your
kernel sources installed.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> >>>
> >>>> This patchset implements dynamic pegging of kconfig symbol
> >>>> into driver modinfo section
> >>>
> >>> First a little bit of motivation here helps, so let me try to
> >>> help fill in some gaps. This may help explain what you have
> >>> been working on a bit more.
> >>>
> >>> First, for those that were not Cc'd but curious about this work
> >>> so far, you can read the patches here:
> >>>
> >>> Original cover letter:
> >>>
> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> >>>
> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> >>>
> >>> There are a few situations in which you may want to extract a
> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
> >>> only considering modules:
> >>>
> >>>  a) When optimizing build requirements for a kernel for a system.
> >>>     That is you boot into a distro kernel and then want to build
> >>>     a slim kernel only with sensible kernel configuration options.
> >>>
> >>>  b) When you are on a distribution kernel but the distribution
> >>>     kernel provided lacks hardware support for your device, you
> >>>     may either want to upgrade the full kernel in which case you
> >>>     want to do a) or -- you may want to just a backports release
> >>>     which provides just the modules you need, you'd use it on top
> >>>     of the distribution kernel.
> >>
> >> c) Having the mapping in sysfs would allow to simplify
> >> streamline_config.pl avoid parsing Makefiles in perl.
> > 
> > Indeed, this was actually a side possible goal here :) only it seems
> > we totally forgot about it while addressing the work.
> > 
> >> Only if the patch did not depend on streamline_config.pl :)
> > 
> > Indeed, lets get rid of that dependency.
> > 
> >> One idea would be to generate
> >> the Module.ksymb in a similar way we generate the modules.builtin file:
> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> >> replaced with
> >>
> >> CONFIG_FOO=m-CONFIG_FOO
> >>
> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
>                                        should be obj-m-CONFIG_%
> 
> >> create the mapping. This would also properly cover cases where we build
> >> the $(obj-m) list from another list. It would certainly create other
> >> corner cases, but it's worth trying IMO.
> > 
> > Neat, would that cover lib-m then as well or other targets that would generate 
> > modules ?
> 
> This would have to be handled as well by the patch. Obviously it's not
> going to be as simple as described in a single paragraph of text :).

Ah OK, it was not clear to me if your suggestion captured all modules already.
There may still be a way. We just have to look harder. Then what precise
target was used would be irrelevant.

> >> Another thing is that we do not necessarily need to record this
> >> information in .modinfo, but we can generate a list in
> >> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> >> that are builtin in the current configuration.
> > 
> > Well so one future use would be for instance a 'make udevconfig' which
> > in theory should be able then to scrape all possible related needed
> > device CONFIG_'s and then give you an optimal kernel configuration.
> > It'd be nice to be able to extract this from the kernel and modules
> > without needing build sources in /lib/modules/`uname -r`/.
> 
> I meant that the file can be installed by modules_install. Like you have
> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
> distro packages.

Got it.

> But really, it's a minor issue how the information is
> presented to userspace.

To us perhaps, but to userspace and users this is a big deal. It'd be nice
to be able to construct a kernel config only with what CONFIG's your hardware
needs, doing this should mean trying to expose enough information in a way
userspace is easily able to handle. The reason to go with the mod section stuff
was this was already being picked up by udevadm for its own hardware db info,
so this seemed like a logical evolution, and it would not require your
kernel sources installed.

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Naveen Kumar]

unsubscribe backports

On 8/24/16, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
>> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
>> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>> >>>
>> >>>> This patchset implements dynamic pegging of kconfig symbol
>> >>>> into driver modinfo section
>> >>>
>> >>> First a little bit of motivation here helps, so let me try to
>> >>> help fill in some gaps. This may help explain what you have
>> >>> been working on a bit more.
>> >>>
>> >>> First, for those that were not Cc'd but curious about this work
>> >>> so far, you can read the patches here:
>> >>>
>> >>> Original cover letter:
>> >>>
>> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>> >>>
>> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>> >>>
>> >>> There are a few situations in which you may want to extract a
>> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> >>> only considering modules:
>> >>>
>> >>>  a) When optimizing build requirements for a kernel for a system.
>> >>>     That is you boot into a distro kernel and then want to build
>> >>>     a slim kernel only with sensible kernel configuration options.
>> >>>
>> >>>  b) When you are on a distribution kernel but the distribution
>> >>>     kernel provided lacks hardware support for your device, you
>> >>>     may either want to upgrade the full kernel in which case you
>> >>>     want to do a) or -- you may want to just a backports release
>> >>>     which provides just the modules you need, you'd use it on top
>> >>>     of the distribution kernel.
>> >>
>> >> c) Having the mapping in sysfs would allow to simplify
>> >> streamline_config.pl avoid parsing Makefiles in perl.
>> >
>> > Indeed, this was actually a side possible goal here :) only it seems
>> > we totally forgot about it while addressing the work.
>> >
>> >> Only if the patch did not depend on streamline_config.pl :)
>> >
>> > Indeed, lets get rid of that dependency.
>> >
>> >> One idea would be to generate
>> >> the Module.ksymb in a similar way we generate the modules.builtin
>> >> file:
>> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> >> replaced with
>> >>
>> >> CONFIG_FOO=m-CONFIG_FOO
>> >>
>> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES))
>> >> to
>>                                        should be obj-m-CONFIG_%
>>
>> >> create the mapping. This would also properly cover cases where we
>> >> build
>> >> the $(obj-m) list from another list. It would certainly create other
>> >> corner cases, but it's worth trying IMO.
>> >
>> > Neat, would that cover lib-m then as well or other targets that would
>> > generate
>> > modules ?
>>
>> This would have to be handled as well by the patch. Obviously it's not
>> going to be as simple as described in a single paragraph of text :).
>
> Ah OK, it was not clear to me if your suggestion captured all modules
> already.
> There may still be a way. We just have to look harder. Then what precise
> target was used would be irrelevant.
>
>> >> Another thing is that we do not necessarily need to record this
>> >> information in .modinfo, but we can generate a list in
>> >> /lib/modules/`uname -r`/ for consumption. It could also include
>> >> drivers
>> >> that are builtin in the current configuration.
>> >
>> > Well so one future use would be for instance a 'make udevconfig' which
>> > in theory should be able then to scrape all possible related needed
>> > device CONFIG_'s and then give you an optimal kernel configuration.
>> > It'd be nice to be able to extract this from the kernel and modules
>> > without needing build sources in /lib/modules/`uname -r`/.
>>
>> I meant that the file can be installed by modules_install. Like you have
>> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
>> distro packages.
>
> Got it.
>
>> But really, it's a minor issue how the information is
>> presented to userspace.
>
> To us perhaps, but to userspace and users this is a big deal. It'd be nice
> to be able to construct a kernel config only with what CONFIG's your
> hardware
> needs, doing this should mean trying to expose enough information in a way
> userspace is easily able to handle. The reason to go with the mod section
> stuff
> was this was already being picked up by udevadm for its own hardware db
> info,
> so this seemed like a logical evolution, and it would not require your
> kernel sources installed.
>
>   Luis
> --
> To unsubscribe from this list: send the line "unsubscribe backports" in
>
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Naveen Kumar]

unsubscribe backports

On 8/24/16, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
>> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
>> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>> >>>
>> >>>> This patchset implements dynamic pegging of kconfig symbol
>> >>>> into driver modinfo section
>> >>>
>> >>> First a little bit of motivation here helps, so let me try to
>> >>> help fill in some gaps. This may help explain what you have
>> >>> been working on a bit more.
>> >>>
>> >>> First, for those that were not Cc'd but curious about this work
>> >>> so far, you can read the patches here:
>> >>>
>> >>> Original cover letter:
>> >>>
>> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>> >>>
>> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>> >>>
>> >>> There are a few situations in which you may want to extract a
>> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> >>> only considering modules:
>> >>>
>> >>>  a) When optimizing build requirements for a kernel for a system.
>> >>>     That is you boot into a distro kernel and then want to build
>> >>>     a slim kernel only with sensible kernel configuration options.
>> >>>
>> >>>  b) When you are on a distribution kernel but the distribution
>> >>>     kernel provided lacks hardware support for your device, you
>> >>>     may either want to upgrade the full kernel in which case you
>> >>>     want to do a) or -- you may want to just a backports release
>> >>>     which provides just the modules you need, you'd use it on top
>> >>>     of the distribution kernel.
>> >>
>> >> c) Having the mapping in sysfs would allow to simplify
>> >> streamline_config.pl avoid parsing Makefiles in perl.
>> >
>> > Indeed, this was actually a side possible goal here :) only it seems
>> > we totally forgot about it while addressing the work.
>> >
>> >> Only if the patch did not depend on streamline_config.pl :)
>> >
>> > Indeed, lets get rid of that dependency.
>> >
>> >> One idea would be to generate
>> >> the Module.ksymb in a similar way we generate the modules.builtin
>> >> file:
>> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> >> replaced with
>> >>
>> >> CONFIG_FOO=m-CONFIG_FOO
>> >>
>> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES))
>> >> to
>>                                        should be obj-m-CONFIG_%
>>
>> >> create the mapping. This would also properly cover cases where we
>> >> build
>> >> the $(obj-m) list from another list. It would certainly create other
>> >> corner cases, but it's worth trying IMO.
>> >
>> > Neat, would that cover lib-m then as well or other targets that would
>> > generate
>> > modules ?
>>
>> This would have to be handled as well by the patch. Obviously it's not
>> going to be as simple as described in a single paragraph of text :).
>
> Ah OK, it was not clear to me if your suggestion captured all modules
> already.
> There may still be a way. We just have to look harder. Then what precise
> target was used would be irrelevant.
>
>> >> Another thing is that we do not necessarily need to record this
>> >> information in .modinfo, but we can generate a list in
>> >> /lib/modules/`uname -r`/ for consumption. It could also include
>> >> drivers
>> >> that are builtin in the current configuration.
>> >
>> > Well so one future use would be for instance a 'make udevconfig' which
>> > in theory should be able then to scrape all possible related needed
>> > device CONFIG_'s and then give you an optimal kernel configuration.
>> > It'd be nice to be able to extract this from the kernel and modules
>> > without needing build sources in /lib/modules/`uname -r`/.
>>
>> I meant that the file can be installed by modules_install. Like you have
>> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
>> distro packages.
>
> Got it.
>
>> But really, it's a minor issue how the information is
>> presented to userspace.
>
> To us perhaps, but to userspace and users this is a big deal. It'd be nice
> to be able to construct a kernel config only with what CONFIG's your
> hardware
> needs, doing this should mean trying to expose enough information in a way
> userspace is easily able to handle. The reason to go with the mod section
> stuff
> was this was already being picked up by udevadm for its own hardware db
> info,
> so this seemed like a logical evolution, and it would not require your
> kernel sources installed.
>
>   Luis
> --
> To unsubscribe from this list: send the line "unsubscribe backports" in
>

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Cristina-Gabriela Moraru]

2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
>
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
>
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
>
> Original cover letter:
>
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
>
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
>
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.
>
> Are there other *uses* outside of the two above for having a direct
> deterministic mapping between module and a kconfig symbol ? What
> if we could also do this for built-in code as well ? kconfig-sat [0] is
> work that is still being ironed out, but if that was in place and we
> handed it a set of required options we want, it should in theory help
> resolve the dependencies to give us a functional .config for a kernel.
> This of course is a long term possible goal just used as an example
> -- but are there other uses?
>
I can't think of any examples. I will add this ones in the cover
letter as motivation.

> Another possible gain here is if addressing a deterministic mapping might
> help with some of the limitations observed in practice over years with
> kconfig and semantics for keconfig. I go into some of this further below.
>
> There are not many options currently available to optimize kernel builds this
> way for modules or choose only what kernel kconfig options you need.  One of
> the best known efforts is the upstream 'make localmodconfig' added by Steven
> years ago to help with a). This has its own set of limitations though. For
> backports we also need to address possible kconfig symbol name changes, and the
> fact that you may boot a box without a driver loaded and as such not be sure
> what new kernel kconfig symbol you may need.
>
> There are shared issues with both goals, addressing shared issues
> can ultimately help both. One of the shared known issues is the
> lack of a deterministic resolution of a module to a respective
> kconfig symbol. Even answering if this is possible or should be
> possible requires some work.
>
> The work Cristina has done here for this year's Google Summer of
> Code is try to help start looking into addressing the question if
> this is possible and then start looking into how we could annotate
> such a thing, and later how we'd use it.
>
> [0] https://kernelnewbies.org/KernelProjects/kconfig-sat
>
>> * updates streamline_config.pl to generate the auxiliary file
>> scripts/mod/Module.ksymb containing associations of driver file
>> names and corresponding kconfig symbols CONFIG_*
>
> Here you make use of existing heuristics -- note however that
> heuristics are needed as we have no direct mapping available
> nor do we have any rules over how to make such a direct mapping
> possible if we wanted this mapping to be deterministic.
>
>> * updates Makefiles to trigger streamline_config.pl call for
>> Module.ksymb generation and pass the determined CONFIG_* symbol
>> as compilation parameter -D in KBUILD_KSYMB variable
>
> Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
> to the build system a config symbol associated with a module, its the right
> approach, however re-using streamline_config.pl seems questionable here, and it
> seems an optimization might be / should be possible. Likewise we'd need to also
> still look into and address *why* some modules don't end up with a respective
> kconfig symbol -- and if we wanted to do something about this. Given the stated
> goals more importantly would be what the costs are to have a deterministic
> mapping.
>

Yes. I've noticed. I will change it completely.

>> * adds kconfig_symb as module attribute
>
> This seems lightweight and follows the logic of the other attributes.
> If its too much, this can be a configurable feature.
>
>> * updates modpost to set KBUILD_KSYMB macro as value for
>> kbuild_symb attribute
>
> Neat.
>
>> Note: the content of the file Module.ksymb is generated for
>> all modules that have only one associate CONFIG option. All
>> others are considered to be components linked at the final
>> modules but not final modules itselves.
>
> To be clear, you only peg the CONFIG_ option if you are
> 100% certain of having a direct mapping, is that right ?
>

Yes. And there are some drivers which have duplicate values in the hash.
foo.o -> CONFIG_FOO CONFIG_FOO

Although this one has one match it's filtered as having 2 CONFIGs.

>> The result of this patchset is the following. All modules from
>> /sys expose the correct CONFIG_* symbol in the module attribute
>> kconfig_symb with some exceptions. For a total number of 58
>> modules,
>
> What kernel configuration did you use ? What architecture ?
> If you use allmodconfig, what do the numbers look like ?
>

I have a virtual machine in which I created the configuration with
make localmodconfig and installed the new kernel.
$  uname -a
Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
x86_64 x86_64 GNU/Linux

Should I send you the .config file ?

>> 4 of them do not:
>>
>> snd_seq_midi_event
>
> Takashi, any reason for this to not have a Kconfig symbol ?
> If we don't want one, is there perhaps any annotation we
> could use to help make a deterministic association if a
> respective kconfig symbol is needed ?
>
>> mptscsih
>
> Same thing -- for both I suppose we can infer that these
> are just helpers, and can / should be enabled if any of
> the devices are found to be present / required.
>
> I can imagine that in certain situations a helper like
> the above might really just be optional rather than required,
> but I'd hope such an optional nature should be reflected via
> Kconfig, the default setting being the recommended option to
> assume as safe then, unless input is provided to help fill
> in the gap and disable / force enable certain options.
>
>> libahci
>
> In this case libahci is spinkled over a lot of lines on the Makefile,
> its a module though, and there is no direct association of dependency
> relationship other than the Makefile association. Using a kconfig
> symbol would replace the Makefile clutter but make the assocation
> and requirement of the dependency explicit through Kconfig. Is
> that desirable or should we just make the assumption that in these
> cases we could assume one Kconfig symbol that adds a module implicitly
> is equivalent as if the module had a kconfig symbol and we had done
> "select foo" for each ? Note though that select is heavy handed [1] --
> and its over use can lead to issues. For instance its known that
> select does not check for dependencies, its a heavy handed way of
> enabling options needed by architectures. So using select here may
> not be desirable. Using "depends on" here would work but it can hide
> from the menu drivers until you have enabled its dependency.
>
> [1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e
>
>> mptbase
>
> Same situation as  mptscsih.
>
>> After a short research:
>>
>> For mptscsih - ./drivers/message/fusion/Makefile
>> obj-$(CONFIG_FUSION_SPI)      += mptbase.o mptscsih.o mptspi.o
>> obj-$(CONFIG_FUSION_FC)               += mptbase.o mptscsih.o mptfc.o
>> obj-$(CONFIG_FUSION_SAS)      += mptbase.o mptscsih.o mptsas.o
>> As appears in the Makefile mptscsi is part of more config
>> options so it's excluded because it's not considered a module
>> by itself but, somehow gets compiled as module. Same for the
>> mptbase. I still have to understand better the Kbuild logic so
>> I cannot state that this is loose practice.
>
> Another reason for this loose use could also be that using "select"
> in Kconfig is heavy handed and can lead to recursive issues, refer
> to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
> dependency limitation / resolution") for more information and
> a clear example of the issue. There is also a list of commits
> with possible resolutions to issues, so that can be inspected further
> to try to understand if perhaps some of the lack of kconfig symbols
> and respective deterministic mapping from module to kconfig is due
> to prior issues with select. I doubt it given that otherwise we'd
> have at least symbols for these -- but worth taking into consideration
> *if* we do then want to add symbols for them, we do not want to lead
> to recursive issues if we add these new symbols.
>
> If we add a kconfig symbol for modules lacking them, what is a
> strategical way to do so in such a way we avoid known past
> issues but also paves the way for the future ?
>
>> For libahci there are multiple CONFIG_ symbols for different
>> architectures.
>>
>> Although past discussion recommended not to use streamline_config,
>> this patchset still relies on it, for the sake of the proof of
>> concept.
>
> OK.
>
>> Although the result is close to the target it still does
>> not provide complete correctness. However it can be replaced by
>> creating another script
>
> Or C code.
>
>> which tries to determine better the
>> module <-> CONFIG associations and output them in auxiliary file
>> Module.ksymb. Maybe this way we could also determine all CONFIGs
>> for a particular driver, not only the exact one that enables it.
>>
>> The auxiliary file is necessary because the Makefile itself does
>> not have a mapping.
>
> Clarification: at least not an easy O(1) quick mapping.
>
>> The makefile includes the config file with
>> directive include which creates a series of internal variables
>> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
>> other Makefiles, lines like
>>
>> obj-$(CONFIG_FOO) = foo.o
>>
>> are resolved in place to obj-y, obj-m etc according to 'make'
>> logic and the association is lost.
>>
>> Finally, what does this patchset provide is an infrastructure
>> to dinamically peg CONFIG_* options to associate drivers using
>> the mapping from Module.ksymb file. Generation of Module.ksymb
>> can be replaced but keeping the same format permit the usage of
>> the other patches.
>
> OK this makes sense, so as it stands it seems we need a better
> way to generate the Module.ksymb and then also address those
> modules which lack a current deterministic mapping, and address
> a resolution with the community on that.
>
>> This patchset is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> Good stuff!
>
>   Luis
>
>> Cristina Moraru (5):
>>   Add generation of Module.symb in streamline_config
>>   Add CONFIG symbol to module as compilation parameter
>>   Trigger Module.ksymb generation in Makefile
>>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>>   Add kconf_symb as kernel module attribute
>>
>>  Makefile                             |  4 ++++
>>  include/linux/module.h               |  1 +
>>  kernel/module.c                      |  2 ++
>>  scripts/Makefile.lib                 |  9 ++++++++-
>>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>>  scripts/mod/modpost.c                |  7 +++++++
>>  6 files changed, 51 insertions(+), 2 deletions(-)
>>
>> --
>> 2.7.4
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Cristina-Gabriela Moraru]

2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
>
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
>
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
>
> Original cover letter:
>
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
>
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
>
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.
>
> Are there other *uses* outside of the two above for having a direct
> deterministic mapping between module and a kconfig symbol ? What
> if we could also do this for built-in code as well ? kconfig-sat [0] is
> work that is still being ironed out, but if that was in place and we
> handed it a set of required options we want, it should in theory help
> resolve the dependencies to give us a functional .config for a kernel.
> This of course is a long term possible goal just used as an example
> -- but are there other uses?
>
I can't think of any examples. I will add this ones in the cover
letter as motivation.

> Another possible gain here is if addressing a deterministic mapping might
> help with some of the limitations observed in practice over years with
> kconfig and semantics for keconfig. I go into some of this further below.
>
> There are not many options currently available to optimize kernel builds this
> way for modules or choose only what kernel kconfig options you need.  One of
> the best known efforts is the upstream 'make localmodconfig' added by Steven
> years ago to help with a). This has its own set of limitations though. For
> backports we also need to address possible kconfig symbol name changes, and the
> fact that you may boot a box without a driver loaded and as such not be sure
> what new kernel kconfig symbol you may need.
>
> There are shared issues with both goals, addressing shared issues
> can ultimately help both. One of the shared known issues is the
> lack of a deterministic resolution of a module to a respective
> kconfig symbol. Even answering if this is possible or should be
> possible requires some work.
>
> The work Cristina has done here for this year's Google Summer of
> Code is try to help start looking into addressing the question if
> this is possible and then start looking into how we could annotate
> such a thing, and later how we'd use it.
>
> [0] https://kernelnewbies.org/KernelProjects/kconfig-sat
>
>> * updates streamline_config.pl to generate the auxiliary file
>> scripts/mod/Module.ksymb containing associations of driver file
>> names and corresponding kconfig symbols CONFIG_*
>
> Here you make use of existing heuristics -- note however that
> heuristics are needed as we have no direct mapping available
> nor do we have any rules over how to make such a direct mapping
> possible if we wanted this mapping to be deterministic.
>
>> * updates Makefiles to trigger streamline_config.pl call for
>> Module.ksymb generation and pass the determined CONFIG_* symbol
>> as compilation parameter -D in KBUILD_KSYMB variable
>
> Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
> to the build system a config symbol associated with a module, its the right
> approach, however re-using streamline_config.pl seems questionable here, and it
> seems an optimization might be / should be possible. Likewise we'd need to also
> still look into and address *why* some modules don't end up with a respective
> kconfig symbol -- and if we wanted to do something about this. Given the stated
> goals more importantly would be what the costs are to have a deterministic
> mapping.
>

Yes. I've noticed. I will change it completely.

>> * adds kconfig_symb as module attribute
>
> This seems lightweight and follows the logic of the other attributes.
> If its too much, this can be a configurable feature.
>
>> * updates modpost to set KBUILD_KSYMB macro as value for
>> kbuild_symb attribute
>
> Neat.
>
>> Note: the content of the file Module.ksymb is generated for
>> all modules that have only one associate CONFIG option. All
>> others are considered to be components linked at the final
>> modules but not final modules itselves.
>
> To be clear, you only peg the CONFIG_ option if you are
> 100% certain of having a direct mapping, is that right ?
>

Yes. And there are some drivers which have duplicate values in the hash.
foo.o -> CONFIG_FOO CONFIG_FOO

Although this one has one match it's filtered as having 2 CONFIGs.

>> The result of this patchset is the following. All modules from
>> /sys expose the correct CONFIG_* symbol in the module attribute
>> kconfig_symb with some exceptions. For a total number of 58
>> modules,
>
> What kernel configuration did you use ? What architecture ?
> If you use allmodconfig, what do the numbers look like ?
>

I have a virtual machine in which I created the configuration with
make localmodconfig and installed the new kernel.
$  uname -a
Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
x86_64 x86_64 GNU/Linux

Should I send you the .config file ?

>> 4 of them do not:
>>
>> snd_seq_midi_event
>
> Takashi, any reason for this to not have a Kconfig symbol ?
> If we don't want one, is there perhaps any annotation we
> could use to help make a deterministic association if a
> respective kconfig symbol is needed ?
>
>> mptscsih
>
> Same thing -- for both I suppose we can infer that these
> are just helpers, and can / should be enabled if any of
> the devices are found to be present / required.
>
> I can imagine that in certain situations a helper like
> the above might really just be optional rather than required,
> but I'd hope such an optional nature should be reflected via
> Kconfig, the default setting being the recommended option to
> assume as safe then, unless input is provided to help fill
> in the gap and disable / force enable certain options.
>
>> libahci
>
> In this case libahci is spinkled over a lot of lines on the Makefile,
> its a module though, and there is no direct association of dependency
> relationship other than the Makefile association. Using a kconfig
> symbol would replace the Makefile clutter but make the assocation
> and requirement of the dependency explicit through Kconfig. Is
> that desirable or should we just make the assumption that in these
> cases we could assume one Kconfig symbol that adds a module implicitly
> is equivalent as if the module had a kconfig symbol and we had done
> "select foo" for each ? Note though that select is heavy handed [1] --
> and its over use can lead to issues. For instance its known that
> select does not check for dependencies, its a heavy handed way of
> enabling options needed by architectures. So using select here may
> not be desirable. Using "depends on" here would work but it can hide
> from the menu drivers until you have enabled its dependency.
>
> [1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e
>
>> mptbase
>
> Same situation as  mptscsih.
>
>> After a short research:
>>
>> For mptscsih - ./drivers/message/fusion/Makefile
>> obj-$(CONFIG_FUSION_SPI)      += mptbase.o mptscsih.o mptspi.o
>> obj-$(CONFIG_FUSION_FC)               += mptbase.o mptscsih.o mptfc.o
>> obj-$(CONFIG_FUSION_SAS)      += mptbase.o mptscsih.o mptsas.o
>> As appears in the Makefile mptscsi is part of more config
>> options so it's excluded because it's not considered a module
>> by itself but, somehow gets compiled as module. Same for the
>> mptbase. I still have to understand better the Kbuild logic so
>> I cannot state that this is loose practice.
>
> Another reason for this loose use could also be that using "select"
> in Kconfig is heavy handed and can lead to recursive issues, refer
> to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
> dependency limitation / resolution") for more information and
> a clear example of the issue. There is also a list of commits
> with possible resolutions to issues, so that can be inspected further
> to try to understand if perhaps some of the lack of kconfig symbols
> and respective deterministic mapping from module to kconfig is due
> to prior issues with select. I doubt it given that otherwise we'd
> have at least symbols for these -- but worth taking into consideration
> *if* we do then want to add symbols for them, we do not want to lead
> to recursive issues if we add these new symbols.
>
> If we add a kconfig symbol for modules lacking them, what is a
> strategical way to do so in such a way we avoid known past
> issues but also paves the way for the future ?
>
>> For libahci there are multiple CONFIG_ symbols for different
>> architectures.
>>
>> Although past discussion recommended not to use streamline_config,
>> this patchset still relies on it, for the sake of the proof of
>> concept.
>
> OK.
>
>> Although the result is close to the target it still does
>> not provide complete correctness. However it can be replaced by
>> creating another script
>
> Or C code.
>
>> which tries to determine better the
>> module <-> CONFIG associations and output them in auxiliary file
>> Module.ksymb. Maybe this way we could also determine all CONFIGs
>> for a particular driver, not only the exact one that enables it.
>>
>> The auxiliary file is necessary because the Makefile itself does
>> not have a mapping.
>
> Clarification: at least not an easy O(1) quick mapping.
>
>> The makefile includes the config file with
>> directive include which creates a series of internal variables
>> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
>> other Makefiles, lines like
>>
>> obj-$(CONFIG_FOO) = foo.o
>>
>> are resolved in place to obj-y, obj-m etc according to 'make'
>> logic and the association is lost.
>>
>> Finally, what does this patchset provide is an infrastructure
>> to dinamically peg CONFIG_* options to associate drivers using
>> the mapping from Module.ksymb file. Generation of Module.ksymb
>> can be replaced but keeping the same format permit the usage of
>> the other patches.
>
> OK this makes sense, so as it stands it seems we need a better
> way to generate the Module.ksymb and then also address those
> modules which lack a current deterministic mapping, and address
> a resolution with the community on that.
>
>> This patchset is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> Good stuff!
>
>   Luis
>
>> Cristina Moraru (5):
>>   Add generation of Module.symb in streamline_config
>>   Add CONFIG symbol to module as compilation parameter
>>   Trigger Module.ksymb generation in Makefile
>>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>>   Add kconf_symb as kernel module attribute
>>
>>  Makefile                             |  4 ++++
>>  include/linux/module.h               |  1 +
>>  kernel/module.c                      |  2 ++
>>  scripts/Makefile.lib                 |  9 ++++++++-
>>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>>  scripts/mod/modpost.c                |  7 +++++++
>>  6 files changed, 51 insertions(+), 2 deletions(-)
>>
>> --
>> 2.7.4
>>
>>

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Mon, Aug 22, 2016 at 09:35:47PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> >
> >> This patchset implements dynamic pegging of kconfig symbol
> >> into driver modinfo section
> >
> > First a little bit of motivation here helps, so let me try to
> > help fill in some gaps. This may help explain what you have
> > been working on a bit more.
> >
> > First, for those that were not Cc'd but curious about this work
> > so far, you can read the patches here:
> >
> > Original cover letter:
> >
> > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> >
> > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> >
> > There are a few situations in which you may want to extract a
> > subset of Kconfig CONFIG_* symbols for a system. At least when
> > only considering modules:
> >
> >  a) When optimizing build requirements for a kernel for a system.
> >     That is you boot into a distro kernel and then want to build
> >     a slim kernel only with sensible kernel configuration options.
> >
> >  b) When you are on a distribution kernel but the distribution
> >     kernel provided lacks hardware support for your device, you
> >     may either want to upgrade the full kernel in which case you
> >     want to do a) or -- you may want to just a backports release
> >     which provides just the modules you need, you'd use it on top
> >     of the distribution kernel.
> >
> > Are there other *uses* outside of the two above for having a direct
> > deterministic mapping between module and a kconfig symbol ? What
> > if we could also do this for built-in code as well ? kconfig-sat [0] is
> > work that is still being ironed out, but if that was in place and we
> > handed it a set of required options we want, it should in theory help
> > resolve the dependencies to give us a functional .config for a kernel.
> > This of course is a long term possible goal just used as an example
> > -- but are there other uses?
> >
> I can't think of any examples. I will add this ones in the cover
> letter as motivation.

Michal provided one more :) -- which was actually a side goal you had I think.

Very clearly documenting the motivation is very important here, side benefits
that you can think of are (some listed above) should help understand why
we may went to or not do some further enhancements to build system to help
with more deterministic backwards mappings.

> > To be clear, you only peg the CONFIG_ option if you are
> > 100% certain of having a direct mapping, is that right ?
> >
> 
> Yes.

OK but as you clarified in other e-mail then the attribute is added but
its just empty. This is rather ambiguous -- lets instead try to not add
the attribute if one is not available.

> And there are some drivers which have duplicate values in the hash.
> foo.o -> CONFIG_FOO CONFIG_FOO

What is the reason for this ?

> Although this one has one match it's filtered as having 2 CONFIGs.
> 
> >> The result of this patchset is the following. All modules from
> >> /sys expose the correct CONFIG_* symbol in the module attribute
> >> kconfig_symb with some exceptions. For a total number of 58
> >> modules,
> >
> > What kernel configuration did you use ? What architecture ?
> > If you use allmodconfig, what do the numbers look like ?
> >
> 
> I have a virtual machine in which I created the configuration with
> make localmodconfig and installed the new kernel.
> $  uname -a
> Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
> x86_64 x86_64 GNU/Linux
> 
> Should I send you the .config file ?

This is not needed, I ran your patches through 0-day and it was happy.
Good job !

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Mon, Aug 22, 2016 at 09:35:47PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> >
> >> This patchset implements dynamic pegging of kconfig symbol
> >> into driver modinfo section
> >
> > First a little bit of motivation here helps, so let me try to
> > help fill in some gaps. This may help explain what you have
> > been working on a bit more.
> >
> > First, for those that were not Cc'd but curious about this work
> > so far, you can read the patches here:
> >
> > Original cover letter:
> >
> > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> >
> > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> >
> > There are a few situations in which you may want to extract a
> > subset of Kconfig CONFIG_* symbols for a system. At least when
> > only considering modules:
> >
> >  a) When optimizing build requirements for a kernel for a system.
> >     That is you boot into a distro kernel and then want to build
> >     a slim kernel only with sensible kernel configuration options.
> >
> >  b) When you are on a distribution kernel but the distribution
> >     kernel provided lacks hardware support for your device, you
> >     may either want to upgrade the full kernel in which case you
> >     want to do a) or -- you may want to just a backports release
> >     which provides just the modules you need, you'd use it on top
> >     of the distribution kernel.
> >
> > Are there other *uses* outside of the two above for having a direct
> > deterministic mapping between module and a kconfig symbol ? What
> > if we could also do this for built-in code as well ? kconfig-sat [0] is
> > work that is still being ironed out, but if that was in place and we
> > handed it a set of required options we want, it should in theory help
> > resolve the dependencies to give us a functional .config for a kernel.
> > This of course is a long term possible goal just used as an example
> > -- but are there other uses?
> >
> I can't think of any examples. I will add this ones in the cover
> letter as motivation.

Michal provided one more :) -- which was actually a side goal you had I think.

Very clearly documenting the motivation is very important here, side benefits
that you can think of are (some listed above) should help understand why
we may went to or not do some further enhancements to build system to help
with more deterministic backwards mappings.

> > To be clear, you only peg the CONFIG_ option if you are
> > 100% certain of having a direct mapping, is that right ?
> >
> 
> Yes.

OK but as you clarified in other e-mail then the attribute is added but
its just empty. This is rather ambiguous -- lets instead try to not add
the attribute if one is not available.

> And there are some drivers which have duplicate values in the hash.
> foo.o -> CONFIG_FOO CONFIG_FOO

What is the reason for this ?

> Although this one has one match it's filtered as having 2 CONFIGs.
> 
> >> The result of this patchset is the following. All modules from
> >> /sys expose the correct CONFIG_* symbol in the module attribute
> >> kconfig_symb with some exceptions. For a total number of 58
> >> modules,
> >
> > What kernel configuration did you use ? What architecture ?
> > If you use allmodconfig, what do the numbers look like ?
> >
> 
> I have a virtual machine in which I created the configuration with
> make localmodconfig and installed the new kernel.
> $  uname -a
> Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
> x86_64 x86_64 GNU/Linux
> 
> Should I send you the .config file ?

This is not needed, I ran your patches through 0-day and it was happy.
Good job !

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Christoph Hellwig]

The idea seems useful, but I reallt don't like the 'reverse-engineering'
approach.

If we want to this properly from the ground up we should just split out
our CONFIG_ SYMBOLS into

MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
only one, then like a bool)

CONFIG_* - just bool, MODULE_ may depend on it, too.

The other nice thing is that we could probably fold most of the Makefiles
into Kconfig using that methods as well, by listing the objectes required
for a module, e.g.

module NVME_TARGET
	tristate "NVMe Target support"
	depends on BLOCK
	depends on CONFIGFS_FS
	name nvmet
	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
	objects discovery.o

module NVME_TARGET_LOOP
	tristate "NVMe loopback device support"
	depends on BLK_DEV_NVME
	depends on NVME_TARGET
	select NVME_FABRICS
	select SG_POOL
	name nvme-loop
	objects loop.o
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Christoph Hellwig]

The idea seems useful, but I reallt don't like the 'reverse-engineering'
approach.

If we want to this properly from the ground up we should just split out
our CONFIG_ SYMBOLS into

MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
only one, then like a bool)

CONFIG_* - just bool, MODULE_ may depend on it, too.

The other nice thing is that we could probably fold most of the Makefiles
into Kconfig using that methods as well, by listing the objectes required
for a module, e.g.

module NVME_TARGET
	tristate "NVMe Target support"
	depends on BLOCK
	depends on CONFIGFS_FS
	name nvmet
	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
	objects discovery.o

module NVME_TARGET_LOOP
	tristate "NVMe loopback device support"
	depends on BLK_DEV_NVME
	depends on NVME_TARGET
	select NVME_FABRICS
	select SG_POOL
	name nvme-loop
	objects loop.o

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Johannes Berg]

> The other nice thing is that we could probably fold most of the
> Makefiles into Kconfig using that methods as well, by listing the
> objectes required for a module, e.g.
> 
> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 

If this was going to be a thing, then you might also have

config NVME_TARGET_FOO
	bool "NVMe target supports FOO"
	module NVME_TARGET
	objects foo.o

The "module" would be like a "depends on" plus giving the module for
generating the Makefile, and now you can really remove most Makefile
stuff... :)

johannes
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Johannes Berg]

> The other nice thing is that we could probably fold most of the
> Makefiles into Kconfig using that methods as well, by listing the
> objectes required for a module, e.g.
> 
> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 

If this was going to be a thing, then you might also have

config NVME_TARGET_FOO
	bool "NVMe target supports FOO"
	module NVME_TARGET
	objects foo.o

The "module" would be like a "depends on" plus giving the module for
generating the Makefile, and now you can really remove most Makefile
stuff... :)

johannes

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Thu, Aug 25, 2016 at 10:00:20AM +0200, Johannes Berg wrote:
> 
> > The other nice thing is that we could probably fold most of the
> > Makefiles into Kconfig using that methods as well, by listing the
> > objectes required for a module, e.g.
> > 
> > module NVME_TARGET
> > 	tristate "NVMe Target support"
> > 	depends on BLOCK
> > 	depends on CONFIGFS_FS
> > 	name nvmet
> > 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> > 	objects discovery.o
> > 
> 
> If this was going to be a thing, then you might also have
> 
> config NVME_TARGET_FOO
> 	bool "NVMe target supports FOO"
> 	module NVME_TARGET
> 	objects foo.o

You mean this instead of a Makefile:

foo-$(CONFIG_NVME_TARGET_FOO) += foo.o

?

> The "module" would be like a "depends on" plus giving the module for
> generating the Makefile, and now you can really remove most Makefile
> stuff... :)

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Thu, Aug 25, 2016 at 10:00:20AM +0200, Johannes Berg wrote:
> 
> > The other nice thing is that we could probably fold most of the
> > Makefiles into Kconfig using that methods as well, by listing the
> > objectes required for a module, e.g.
> > 
> > module NVME_TARGET
> > 	tristate "NVMe Target support"
> > 	depends on BLOCK
> > 	depends on CONFIGFS_FS
> > 	name nvmet
> > 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> > 	objects discovery.o
> > 
> 
> If this was going to be a thing, then you might also have
> 
> config NVME_TARGET_FOO
> 	bool "NVMe target supports FOO"
> 	module NVME_TARGET
> 	objects foo.o

You mean this instead of a Makefile:

foo-$(CONFIG_NVME_TARGET_FOO) += foo.o

?

> The "module" would be like a "depends on" plus giving the module for
> generating the Makefile, and now you can really remove most Makefile
> stuff... :)

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Michal Marek]

On 2016-08-25 09:43, Christoph Hellwig wrote:
> The idea seems useful, but I reallt don't like the 'reverse-engineering'
> approach.
> 
> If we want to this properly from the ground up we should just split out
> our CONFIG_ SYMBOLS into
> 
> MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> only one, then like a bool)
> 
> CONFIG_* - just bool, MODULE_ may depend on it, too.
> 
> The other nice thing is that we could probably fold most of the Makefiles
> into Kconfig using that methods as well, by listing the objectes required
> for a module, e.g.
> 
> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 
> module NVME_TARGET_LOOP
> 	tristate "NVMe loopback device support"
> 	depends on BLK_DEV_NVME
> 	depends on NVME_TARGET
> 	select NVME_FABRICS
> 	select SG_POOL
> 	name nvme-loop
> 	objects loop.o

It looks nice for the simple case, but as soon as you need something
non-standard, you will need to mix a kconfig generated and a manually
written Makefile snippet.

Michal
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Michal Marek]

On 2016-08-25 09:43, Christoph Hellwig wrote:
> The idea seems useful, but I reallt don't like the 'reverse-engineering'
> approach.
> 
> If we want to this properly from the ground up we should just split out
> our CONFIG_ SYMBOLS into
> 
> MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> only one, then like a bool)
> 
> CONFIG_* - just bool, MODULE_ may depend on it, too.
> 
> The other nice thing is that we could probably fold most of the Makefiles
> into Kconfig using that methods as well, by listing the objectes required
> for a module, e.g.
> 
> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 
> module NVME_TARGET_LOOP
> 	tristate "NVMe loopback device support"
> 	depends on BLK_DEV_NVME
> 	depends on NVME_TARGET
> 	select NVME_FABRICS
> 	select SG_POOL
> 	name nvme-loop
> 	objects loop.o

It looks nice for the simple case, but as soon as you need something
non-standard, you will need to mix a kconfig generated and a manually
written Makefile snippet.

Michal

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Thu, Aug 25, 2016 at 09:43:13AM +0200, Christoph Hellwig wrote:
> The idea seems useful, but I reallt don't like the 'reverse-engineering'
> approach.
> 
> If we want to this properly from the ground up we should just split out
> our CONFIG_ SYMBOLS into
> 
> MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> only one, then like a bool)
> 
> CONFIG_* - just bool, MODULE_ may depend on it, too.

Curious what does the split buy us if the real meaningful input is the value
assigned to the config ? Ie, MODULE_FOO=m would be the modules we want to
check for.

> The other nice thing is that we could probably fold most of the Makefiles
> into Kconfig using that methods as well, by listing the objectes required
> for a module, e.g.

OK If the Kconfig file has the objects listed I can see the gain of using
Kconfig then to more easily map out to a symbol, given doing this on Makefiles
is not straight forward.

> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 
> module NVME_TARGET_LOOP
> 	tristate "NVMe loopback device support"
> 	depends on BLK_DEV_NVME
> 	depends on NVME_TARGET
> 	select NVME_FABRICS
> 	select SG_POOL
> 	name nvme-loop
> 	objects loop.o

I can see a huge win of having a direct specification that provides as a
feature two way mapping from CONFIG <--> module (objects) and backwards again
easily and clearly without hacks, specially if upon boot then we can then
provide the precise kernel configuration you need, for both built-in and
modules. The above could help with modules -- for built-in reverse mapping
we'd need something else, perhaps a configurable option to keep tabs on
inits called with associated configs.

The above would be a pretty intrusive change though, in comparison to
Cristina's original approach. The reverse-engineering object --> config
aspect of her work and of the old scripts/kconfig/streamline_config.pl
explains why it was hard. I'd be curious to learn of other gains possible
other than those listed so far, if we had this.

Re-iterating gains of having a simple two way CONFIG <--> module (objects)
mapping (following the above proposal now):

a) When optimizing build requirements for a kernel for a system.                                                                                                           
   That is you boot into a distro kernel and then want to build                                                                                                            
   a slim kernel only with sensible kernel configuration options.                                                                                                          

b) When you are on a distribution kernel but the distribution                                                                                                              
   kernel provided lacks hardware support for your device, you                                                                                                             
   may either want to upgrade the full kernel in which case you                                                                                                            
   want to do a) or -- you may want to just a backports release                                                                                                            
   which provides just the modules you need, you'd use it on top                                                                                                           
   of the distribution kernel.                                                                                                                                             

c) Having the mapping in sysfs would allow to simplify                                                                                                                       
   streamline_config.pl avoid parsing Makefiles in perl. (From Michal)

d) Fold most of the Makefiles into Kconfig

In retrospect c) still seems related to a) as we'd do away with
the hacks completely needed by streamline_config.pl, a) can be
augmented if we figure out a built-in solution as well.

d) Just seems like collateral of a more precise mapping than
what a Makefile provides. Anything else ?

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis R. Rodriguez]

On Thu, Aug 25, 2016 at 09:43:13AM +0200, Christoph Hellwig wrote:
> The idea seems useful, but I reallt don't like the 'reverse-engineering'
> approach.
> 
> If we want to this properly from the ground up we should just split out
> our CONFIG_ SYMBOLS into
> 
> MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> only one, then like a bool)
> 
> CONFIG_* - just bool, MODULE_ may depend on it, too.

Curious what does the split buy us if the real meaningful input is the value
assigned to the config ? Ie, MODULE_FOO=m would be the modules we want to
check for.

> The other nice thing is that we could probably fold most of the Makefiles
> into Kconfig using that methods as well, by listing the objectes required
> for a module, e.g.

OK If the Kconfig file has the objects listed I can see the gain of using
Kconfig then to more easily map out to a symbol, given doing this on Makefiles
is not straight forward.

> module NVME_TARGET
> 	tristate "NVMe Target support"
> 	depends on BLOCK
> 	depends on CONFIGFS_FS
> 	name nvmet
> 	objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> 	objects discovery.o
> 
> module NVME_TARGET_LOOP
> 	tristate "NVMe loopback device support"
> 	depends on BLK_DEV_NVME
> 	depends on NVME_TARGET
> 	select NVME_FABRICS
> 	select SG_POOL
> 	name nvme-loop
> 	objects loop.o

I can see a huge win of having a direct specification that provides as a
feature two way mapping from CONFIG <--> module (objects) and backwards again
easily and clearly without hacks, specially if upon boot then we can then
provide the precise kernel configuration you need, for both built-in and
modules. The above could help with modules -- for built-in reverse mapping
we'd need something else, perhaps a configurable option to keep tabs on
inits called with associated configs.

The above would be a pretty intrusive change though, in comparison to
Cristina's original approach. The reverse-engineering object --> config
aspect of her work and of the old scripts/kconfig/streamline_config.pl
explains why it was hard. I'd be curious to learn of other gains possible
other than those listed so far, if we had this.

Re-iterating gains of having a simple two way CONFIG <--> module (objects)
mapping (following the above proposal now):

a) When optimizing build requirements for a kernel for a system.                                                                                                           
   That is you boot into a distro kernel and then want to build                                                                                                            
   a slim kernel only with sensible kernel configuration options.                                                                                                          

b) When you are on a distribution kernel but the distribution                                                                                                              
   kernel provided lacks hardware support for your device, you                                                                                                             
   may either want to upgrade the full kernel in which case you                                                                                                            
   want to do a) or -- you may want to just a backports release                                                                                                            
   which provides just the modules you need, you'd use it on top                                                                                                           
   of the distribution kernel.                                                                                                                                             

c) Having the mapping in sysfs would allow to simplify                                                                                                                       
   streamline_config.pl avoid parsing Makefiles in perl. (From Michal)

d) Fold most of the Makefiles into Kconfig

In retrospect c) still seems related to a) as we'd do away with
the hacks completely needed by streamline_config.pl, a) can be
augmented if we figure out a built-in solution as well.

d) Just seems like collateral of a more precise mapping than
what a Makefile provides. Anything else ?

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Thu, Aug 25, 2016 at 2:19 PM Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>
> On Thu, Aug 25, 2016 at 09:43:13AM +0200, Christoph Hellwig wrote:
> > The idea seems useful, but I reallt don't like the 'reverse-engineering'
> > approach.
> >
> > If we want to this properly from the ground up we should just split out
> > our CONFIG_ SYMBOLS into
> >
> > MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> > only one, then like a bool)
> >
> > CONFIG_* - just bool, MODULE_ may depend on it, too.
>
> Curious what does the split buy us if the real meaningful input is the value
> assigned to the config ? Ie, MODULE_FOO=m would be the modules we want to
> check for.

Also, unless you have kernel sources I don't think you could still
easily infer which MODULE_* config option enabled the module, could
you? And this was one of the original goals with this patch set. I
still think extending the modinfo section would be needed to allow
udevadm info to easily extract the information, no?

> > The other nice thing is that we could probably fold most of the Makefiles
> > into Kconfig using that methods as well, by listing the objectes required
> > for a module, e.g.
>
> OK If the Kconfig file has the objects listed I can see the gain of using
> Kconfig then to more easily map out to a symbol, given doing this on Makefiles
> is not straight forward.
>
> > module NVME_TARGET
> >       tristate "NVMe Target support"
> >       depends on BLOCK
> >       depends on CONFIGFS_FS
> >       name nvmet
> >       objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> >       objects discovery.o
> >
> > module NVME_TARGET_LOOP
> >       tristate "NVMe loopback device support"
> >       depends on BLK_DEV_NVME
> >       depends on NVME_TARGET
> >       select NVME_FABRICS
> >       select SG_POOL
> >       name nvme-loop
> >       objects loop.o
>
> I can see a huge win of having a direct specification that provides as a
> feature two way mapping from CONFIG <--> module (objects) and backwards again
> easily and clearly without hacks, specially if upon boot then we can then
> provide the precise kernel configuration you need, for both built-in and
> modules. The above could help with modules -- for built-in reverse mapping
> we'd need something else, perhaps a configurable option to keep tabs on
> inits called with associated configs.
>
> The above would be a pretty intrusive change though, in comparison to
> Cristina's original approach.

I think this is the biggest issue so far, and that we have no code to
showcase for it. Meanwhile pegging at least a kconfig symbol to
modinfo for at least a lot of modules would help considerably for
other use cases mentioned.

> The reverse-engineering object --> config
> aspect of her work and of the old scripts/kconfig/streamline_config.pl
> explains why it was hard. I'd be curious to learn of other gains possible
> other than those listed so far, if we had this.
>
> Re-iterating gains of having a simple two way CONFIG <--> module (objects)
> mapping (following the above proposal now):
>
> a) When optimizing build requirements for a kernel for a system.
>    That is you boot into a distro kernel and then want to build
>    a slim kernel only with sensible kernel configuration options.
>
> b) When you are on a distribution kernel but the distribution
>    kernel provided lacks hardware support for your device, you
>    may either want to upgrade the full kernel in which case you
>    want to do a) or -- you may want to just a backports release
>    which provides just the modules you need, you'd use it on top
>    of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
>    streamline_config.pl avoid parsing Makefiles in perl. (From Michal)

It took me a while to recall where this was and why after 3 years of
not following up on this thread. So in case others fall into the same
boat, the sysfs gain here is that the modinfo section could be
readable then on any module's:

/sys/module/e1000/section/kconfig_symb

> d) Fold most of the Makefiles into Kconfig
>
> In retrospect c) still seems related to a) as we'd do away with
> the hacks completely needed by streamline_config.pl, a) can be
> augmented if we figure out a built-in solution as well.
>
> d) Just seems like collateral of a more precise mapping than
> what a Makefile provides. Anything else ?

In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
on this, I'm inclined to believe *at least* having some kconfig_symb
exposed for some modules is better than nothing. Christoph are you
totally opposed to this effort until we get a non-reverse engineered
effort in place? It just seems like an extraordinary amount of work
and I'm not quite sure who's volunteering to do it.

Other stakeholders may benefit from at least having some config -->
module mapping for now. Not just backports or building slimmer
kernels.

 Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Thu, Aug 25, 2016 at 2:19 PM Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>
> On Thu, Aug 25, 2016 at 09:43:13AM +0200, Christoph Hellwig wrote:
> > The idea seems useful, but I reallt don't like the 'reverse-engineering'
> > approach.
> >
> > If we want to this properly from the ground up we should just split out
> > our CONFIG_ SYMBOLS into
> >
> > MODULE_* - builds exactly one module (tristate, or maybe also as a built-in
> > only one, then like a bool)
> >
> > CONFIG_* - just bool, MODULE_ may depend on it, too.
>
> Curious what does the split buy us if the real meaningful input is the value
> assigned to the config ? Ie, MODULE_FOO=m would be the modules we want to
> check for.

Also, unless you have kernel sources I don't think you could still
easily infer which MODULE_* config option enabled the module, could
you? And this was one of the original goals with this patch set. I
still think extending the modinfo section would be needed to allow
udevadm info to easily extract the information, no?

> > The other nice thing is that we could probably fold most of the Makefiles
> > into Kconfig using that methods as well, by listing the objectes required
> > for a module, e.g.
>
> OK If the Kconfig file has the objects listed I can see the gain of using
> Kconfig then to more easily map out to a symbol, given doing this on Makefiles
> is not straight forward.
>
> > module NVME_TARGET
> >       tristate "NVMe Target support"
> >       depends on BLOCK
> >       depends on CONFIGFS_FS
> >       name nvmet
> >       objects core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o
> >       objects discovery.o
> >
> > module NVME_TARGET_LOOP
> >       tristate "NVMe loopback device support"
> >       depends on BLK_DEV_NVME
> >       depends on NVME_TARGET
> >       select NVME_FABRICS
> >       select SG_POOL
> >       name nvme-loop
> >       objects loop.o
>
> I can see a huge win of having a direct specification that provides as a
> feature two way mapping from CONFIG <--> module (objects) and backwards again
> easily and clearly without hacks, specially if upon boot then we can then
> provide the precise kernel configuration you need, for both built-in and
> modules. The above could help with modules -- for built-in reverse mapping
> we'd need something else, perhaps a configurable option to keep tabs on
> inits called with associated configs.
>
> The above would be a pretty intrusive change though, in comparison to
> Cristina's original approach.

I think this is the biggest issue so far, and that we have no code to
showcase for it. Meanwhile pegging at least a kconfig symbol to
modinfo for at least a lot of modules would help considerably for
other use cases mentioned.

> The reverse-engineering object --> config
> aspect of her work and of the old scripts/kconfig/streamline_config.pl
> explains why it was hard. I'd be curious to learn of other gains possible
> other than those listed so far, if we had this.
>
> Re-iterating gains of having a simple two way CONFIG <--> module (objects)
> mapping (following the above proposal now):
>
> a) When optimizing build requirements for a kernel for a system.
>    That is you boot into a distro kernel and then want to build
>    a slim kernel only with sensible kernel configuration options.
>
> b) When you are on a distribution kernel but the distribution
>    kernel provided lacks hardware support for your device, you
>    may either want to upgrade the full kernel in which case you
>    want to do a) or -- you may want to just a backports release
>    which provides just the modules you need, you'd use it on top
>    of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
>    streamline_config.pl avoid parsing Makefiles in perl. (From Michal)

It took me a while to recall where this was and why after 3 years of
not following up on this thread. So in case others fall into the same
boat, the sysfs gain here is that the modinfo section could be
readable then on any module's:

/sys/module/e1000/section/kconfig_symb

> d) Fold most of the Makefiles into Kconfig
>
> In retrospect c) still seems related to a) as we'd do away with
> the hacks completely needed by streamline_config.pl, a) can be
> augmented if we figure out a built-in solution as well.
>
> d) Just seems like collateral of a more precise mapping than
> what a Makefile provides. Anything else ?

In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
on this, I'm inclined to believe *at least* having some kconfig_symb
exposed for some modules is better than nothing. Christoph are you
totally opposed to this effort until we get a non-reverse engineered
effort in place? It just seems like an extraordinary amount of work
and I'm not quite sure who's volunteering to do it.

Other stakeholders may benefit from at least having some config -->
module mapping for now. Not just backports or building slimmer
kernels.

 Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> on this, I'm inclined to believe *at least* having some kconfig_symb
> exposed for some modules is better than nothing. Christoph are you
> totally opposed to this effort until we get a non-reverse engineered
> effort in place? It just seems like an extraordinary amount of work
> and I'm not quite sure who's volunteering to do it.
>
> Other stakeholders may benefit from at least having some config -->
> module mapping for now. Not just backports or building slimmer
> kernels.

Christoph, *poke*

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> on this, I'm inclined to believe *at least* having some kconfig_symb
> exposed for some modules is better than nothing. Christoph are you
> totally opposed to this effort until we get a non-reverse engineered
> effort in place? It just seems like an extraordinary amount of work
> and I'm not quite sure who's volunteering to do it.
>
> Other stakeholders may benefit from at least having some config -->
> module mapping for now. Not just backports or building slimmer
> kernels.

Christoph, *poke*

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Christoph Hellwig]

On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > on this, I'm inclined to believe *at least* having some kconfig_symb
> > exposed for some modules is better than nothing. Christoph are you
> > totally opposed to this effort until we get a non-reverse engineered
> > effort in place? It just seems like an extraordinary amount of work
> > and I'm not quite sure who's volunteering to do it.
> >
> > Other stakeholders may benefit from at least having some config -->
> > module mapping for now. Not just backports or building slimmer
> > kernels.
> 
> Christoph, *poke*

Yes, I'm still totally opposed to a half-backed hack like this.
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Christoph Hellwig]

On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > on this, I'm inclined to believe *at least* having some kconfig_symb
> > exposed for some modules is better than nothing. Christoph are you
> > totally opposed to this effort until we get a non-reverse engineered
> > effort in place? It just seems like an extraordinary amount of work
> > and I'm not quite sure who's volunteering to do it.
> >
> > Other stakeholders may benefit from at least having some config -->
> > module mapping for now. Not just backports or building slimmer
> > kernels.
> 
> Christoph, *poke*

Yes, I'm still totally opposed to a half-backed hack like this.

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > exposed for some modules is better than nothing. Christoph are you
> > > totally opposed to this effort until we get a non-reverse engineered
> > > effort in place? It just seems like an extraordinary amount of work
> > > and I'm not quite sure who's volunteering to do it.
> > >
> > > Other stakeholders may benefit from at least having some config -->
> > > module mapping for now. Not just backports or building slimmer
> > > kernels.
> >
> > Christoph, *poke*
>
> Yes, I'm still totally opposed to a half-backed hack like this.

The solution puts forward a mechanism to add a kconfig_symb where we
are 100% certain we have a direct module --> config mapping.

This is *currently* determined when the streamline_config.pl finds
that an object has only *one* associated config symbol associated. As
Cristina noted, of 62 modules on a running system 58 of them ended up
getting the kconfig_symb assigned, that is 93.5% of all modules on the
system being tested. For the other modules, if they did want this
association, we could allow a way for modules to define their own
KBUILD_KCONF variable so that this could be considered as well, or
they can look at their own kconfig stuff to try to fit the model that
does work. To be clear, the heuristics *can* be updated if there is
confidence in alternative methods for resolution. But since it is
reflective of our current situation, I cannot consider it a hack.

This implementation is a reflection of our reality in the kernel, and
as has been discussed in this thread, if we want to correct the gaps
we need to do a lot of work. And *no one* is working towards these
goals.

That said, even if you go forward with an intrusive solution like the
one you proposed we could still use the same kconfig_symb...

So no, I don't see this as a hack. It's a reflection as to our current
reality. And I cannot see how the kconfig_symb can lie or be
incorrect. So in fact I think that pushing this forward also makes the
problem statement clearer for the future of what semantics needs to be
addressed, and helps us even annotate the problematic areas of the
kernel.

What negative aspects do you see with this being merged in practice?

 Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > exposed for some modules is better than nothing. Christoph are you
> > > totally opposed to this effort until we get a non-reverse engineered
> > > effort in place? It just seems like an extraordinary amount of work
> > > and I'm not quite sure who's volunteering to do it.
> > >
> > > Other stakeholders may benefit from at least having some config -->
> > > module mapping for now. Not just backports or building slimmer
> > > kernels.
> >
> > Christoph, *poke*
>
> Yes, I'm still totally opposed to a half-backed hack like this.

The solution puts forward a mechanism to add a kconfig_symb where we
are 100% certain we have a direct module --> config mapping.

This is *currently* determined when the streamline_config.pl finds
that an object has only *one* associated config symbol associated. As
Cristina noted, of 62 modules on a running system 58 of them ended up
getting the kconfig_symb assigned, that is 93.5% of all modules on the
system being tested. For the other modules, if they did want this
association, we could allow a way for modules to define their own
KBUILD_KCONF variable so that this could be considered as well, or
they can look at their own kconfig stuff to try to fit the model that
does work. To be clear, the heuristics *can* be updated if there is
confidence in alternative methods for resolution. But since it is
reflective of our current situation, I cannot consider it a hack.

This implementation is a reflection of our reality in the kernel, and
as has been discussed in this thread, if we want to correct the gaps
we need to do a lot of work. And *no one* is working towards these
goals.

That said, even if you go forward with an intrusive solution like the
one you proposed we could still use the same kconfig_symb...

So no, I don't see this as a hack. It's a reflection as to our current
reality. And I cannot see how the kconfig_symb can lie or be
incorrect. So in fact I think that pushing this forward also makes the
problem statement clearer for the future of what semantics needs to be
addressed, and helps us even annotate the problematic areas of the
kernel.

What negative aspects do you see with this being merged in practice?

 Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Greg Kroah-Hartman]

On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> >
> > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > exposed for some modules is better than nothing. Christoph are you
> > > > totally opposed to this effort until we get a non-reverse engineered
> > > > effort in place? It just seems like an extraordinary amount of work
> > > > and I'm not quite sure who's volunteering to do it.
> > > >
> > > > Other stakeholders may benefit from at least having some config -->
> > > > module mapping for now. Not just backports or building slimmer
> > > > kernels.
> > >
> > > Christoph, *poke*
> >
> > Yes, I'm still totally opposed to a half-backed hack like this.
> 
> The solution puts forward a mechanism to add a kconfig_symb where we
> are 100% certain we have a direct module --> config mapping.
> 
> This is *currently* determined when the streamline_config.pl finds
> that an object has only *one* associated config symbol associated. As
> Cristina noted, of 62 modules on a running system 58 of them ended up
> getting the kconfig_symb assigned, that is 93.5% of all modules on the
> system being tested. For the other modules, if they did want this
> association, we could allow a way for modules to define their own
> KBUILD_KCONF variable so that this could be considered as well, or
> they can look at their own kconfig stuff to try to fit the model that
> does work. To be clear, the heuristics *can* be updated if there is
> confidence in alternative methods for resolution. But since it is
> reflective of our current situation, I cannot consider it a hack.
> 
> This implementation is a reflection of our reality in the kernel, and
> as has been discussed in this thread, if we want to correct the gaps
> we need to do a lot of work. And *no one* is working towards these
> goals.
> 
> That said, even if you go forward with an intrusive solution like the
> one you proposed we could still use the same kconfig_symb...
> 
> So no, I don't see this as a hack. It's a reflection as to our current
> reality. And I cannot see how the kconfig_symb can lie or be
> incorrect. So in fact I think that pushing this forward also makes the
> problem statement clearer for the future of what semantics needs to be
> addressed, and helps us even annotate the problematic areas of the
> kernel.
> 
> What negative aspects do you see with this being merged in practice?

I'm trying to see what the actual problem that you are wanting to solve
here with this.  What exactly is it?  Who needs to determine the
"singular" configuration option that caused a kernel module to be built
at the expense of all other options?  What can that be used for and who
will use it?

confused,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Greg Kroah-Hartman]

On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> >
> > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > exposed for some modules is better than nothing. Christoph are you
> > > > totally opposed to this effort until we get a non-reverse engineered
> > > > effort in place? It just seems like an extraordinary amount of work
> > > > and I'm not quite sure who's volunteering to do it.
> > > >
> > > > Other stakeholders may benefit from at least having some config -->
> > > > module mapping for now. Not just backports or building slimmer
> > > > kernels.
> > >
> > > Christoph, *poke*
> >
> > Yes, I'm still totally opposed to a half-backed hack like this.
> 
> The solution puts forward a mechanism to add a kconfig_symb where we
> are 100% certain we have a direct module --> config mapping.
> 
> This is *currently* determined when the streamline_config.pl finds
> that an object has only *one* associated config symbol associated. As
> Cristina noted, of 62 modules on a running system 58 of them ended up
> getting the kconfig_symb assigned, that is 93.5% of all modules on the
> system being tested. For the other modules, if they did want this
> association, we could allow a way for modules to define their own
> KBUILD_KCONF variable so that this could be considered as well, or
> they can look at their own kconfig stuff to try to fit the model that
> does work. To be clear, the heuristics *can* be updated if there is
> confidence in alternative methods for resolution. But since it is
> reflective of our current situation, I cannot consider it a hack.
> 
> This implementation is a reflection of our reality in the kernel, and
> as has been discussed in this thread, if we want to correct the gaps
> we need to do a lot of work. And *no one* is working towards these
> goals.
> 
> That said, even if you go forward with an intrusive solution like the
> one you proposed we could still use the same kconfig_symb...
> 
> So no, I don't see this as a hack. It's a reflection as to our current
> reality. And I cannot see how the kconfig_symb can lie or be
> incorrect. So in fact I think that pushing this forward also makes the
> problem statement clearer for the future of what semantics needs to be
> addressed, and helps us even annotate the problematic areas of the
> kernel.
> 
> What negative aspects do you see with this being merged in practice?

I'm trying to see what the actual problem that you are wanting to solve
here with this.  What exactly is it?  Who needs to determine the
"singular" configuration option that caused a kernel module to be built
at the expense of all other options?  What can that be used for and who
will use it?

confused,

greg k-h

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > >
> > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > and I'm not quite sure who's volunteering to do it.
> > > > >
> > > > > Other stakeholders may benefit from at least having some config -->
> > > > > module mapping for now. Not just backports or building slimmer
> > > > > kernels.
> > > >
> > > > Christoph, *poke*
> > >
> > > Yes, I'm still totally opposed to a half-backed hack like this.
> > 
> > The solution puts forward a mechanism to add a kconfig_symb where we
> > are 100% certain we have a direct module --> config mapping.
> > 
> > This is *currently* determined when the streamline_config.pl finds
> > that an object has only *one* associated config symbol associated. As
> > Cristina noted, of 62 modules on a running system 58 of them ended up
> > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > system being tested. For the other modules, if they did want this
> > association, we could allow a way for modules to define their own
> > KBUILD_KCONF variable so that this could be considered as well, or
> > they can look at their own kconfig stuff to try to fit the model that
> > does work. To be clear, the heuristics *can* be updated if there is
> > confidence in alternative methods for resolution. But since it is
> > reflective of our current situation, I cannot consider it a hack.
> > 
> > This implementation is a reflection of our reality in the kernel, and
> > as has been discussed in this thread, if we want to correct the gaps
> > we need to do a lot of work. And *no one* is working towards these
> > goals.
> > 
> > That said, even if you go forward with an intrusive solution like the
> > one you proposed we could still use the same kconfig_symb...
> > 
> > So no, I don't see this as a hack. It's a reflection as to our current
> > reality. And I cannot see how the kconfig_symb can lie or be
> > incorrect. So in fact I think that pushing this forward also makes the
> > problem statement clearer for the future of what semantics needs to be
> > addressed, and helps us even annotate the problematic areas of the
> > kernel.
> > 
> > What negative aspects do you see with this being merged in practice?
> 
> I'm trying to see what the actual problem that you are wanting to solve
> here with this.  What exactly is it?

The problem is that there is no current maping of a module to respective
kconfig symbol.

> Who needs to determine the
> "singular" configuration option that caused a kernel module to be built
> at the expense of all other options?

Folks wanting to slim down their kernel build, and users of backports.

> What can that be used for and who will use it?

Without a mapping there is no clean way to let you slim down your kernel
using a distro kernel as a base, enabling only those things you really
need.

Historically those interested would just have streamline_config.pl at
their disposal, but it isn't perfect. Although Cristina extended
streamline_config.pl, the proper approach here would be to see if the
heuristics in streamline_config.pl which she made use of to map a config
symbol for a module could be written in C. The safe heuristic used is
that for modules where there was only one kconfig symbol associated,
we'd be certain that kconfig symbol was the one needed for that module.

A side benefit is also that by having a modinfo section for the kconfib
symbol we'd have access to it via sysfs, an example is
/sys/module/e1000/section/kconfig_symb, and then streamline_config.pl
could be simplified further by not having to parse Makefiles in perl.
The reason for using a modinfo section also is you could get the
information about configs needed for modules without requiring your
kernel sources, and this information then also becomes available to
augment udevadm for its own hardware db info.

Another set of users of this would be users of backports. For instance
users on distro kernels but their current kernel lacks updated drivers
for one of your devices, and you want to use a backports release, but
just want to build / only the drivers you really need for the release.
Backports then could easily scrape for the sysfs attribute / udevadm db
to let you build and install only the modules you need.

Long term users are developers working on trying to clear up kconfig
semantics. Having a clear indicator in our tree about modules where a
direct mapping was not possible (say with a KBUILD_KCONF definition in
their Makefiles to clarify which config symbol enables it) gives us an
idea of kconfig entries which may be contrived and could use some
clarifications.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > >
> > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > and I'm not quite sure who's volunteering to do it.
> > > > >
> > > > > Other stakeholders may benefit from at least having some config -->
> > > > > module mapping for now. Not just backports or building slimmer
> > > > > kernels.
> > > >
> > > > Christoph, *poke*
> > >
> > > Yes, I'm still totally opposed to a half-backed hack like this.
> > 
> > The solution puts forward a mechanism to add a kconfig_symb where we
> > are 100% certain we have a direct module --> config mapping.
> > 
> > This is *currently* determined when the streamline_config.pl finds
> > that an object has only *one* associated config symbol associated. As
> > Cristina noted, of 62 modules on a running system 58 of them ended up
> > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > system being tested. For the other modules, if they did want this
> > association, we could allow a way for modules to define their own
> > KBUILD_KCONF variable so that this could be considered as well, or
> > they can look at their own kconfig stuff to try to fit the model that
> > does work. To be clear, the heuristics *can* be updated if there is
> > confidence in alternative methods for resolution. But since it is
> > reflective of our current situation, I cannot consider it a hack.
> > 
> > This implementation is a reflection of our reality in the kernel, and
> > as has been discussed in this thread, if we want to correct the gaps
> > we need to do a lot of work. And *no one* is working towards these
> > goals.
> > 
> > That said, even if you go forward with an intrusive solution like the
> > one you proposed we could still use the same kconfig_symb...
> > 
> > So no, I don't see this as a hack. It's a reflection as to our current
> > reality. And I cannot see how the kconfig_symb can lie or be
> > incorrect. So in fact I think that pushing this forward also makes the
> > problem statement clearer for the future of what semantics needs to be
> > addressed, and helps us even annotate the problematic areas of the
> > kernel.
> > 
> > What negative aspects do you see with this being merged in practice?
> 
> I'm trying to see what the actual problem that you are wanting to solve
> here with this.  What exactly is it?

The problem is that there is no current maping of a module to respective
kconfig symbol.

> Who needs to determine the
> "singular" configuration option that caused a kernel module to be built
> at the expense of all other options?

Folks wanting to slim down their kernel build, and users of backports.

> What can that be used for and who will use it?

Without a mapping there is no clean way to let you slim down your kernel
using a distro kernel as a base, enabling only those things you really
need.

Historically those interested would just have streamline_config.pl at
their disposal, but it isn't perfect. Although Cristina extended
streamline_config.pl, the proper approach here would be to see if the
heuristics in streamline_config.pl which she made use of to map a config
symbol for a module could be written in C. The safe heuristic used is
that for modules where there was only one kconfig symbol associated,
we'd be certain that kconfig symbol was the one needed for that module.

A side benefit is also that by having a modinfo section for the kconfib
symbol we'd have access to it via sysfs, an example is
/sys/module/e1000/section/kconfig_symb, and then streamline_config.pl
could be simplified further by not having to parse Makefiles in perl.
The reason for using a modinfo section also is you could get the
information about configs needed for modules without requiring your
kernel sources, and this information then also becomes available to
augment udevadm for its own hardware db info.

Another set of users of this would be users of backports. For instance
users on distro kernels but their current kernel lacks updated drivers
for one of your devices, and you want to use a backports release, but
just want to build / only the drivers you really need for the release.
Backports then could easily scrape for the sysfs attribute / udevadm db
to let you build and install only the modules you need.

Long term users are developers working on trying to clear up kconfig
semantics. Having a clear indicator in our tree about modules where a
direct mapping was not possible (say with a KBUILD_KCONF definition in
their Makefiles to clarify which config symbol enables it) gives us an
idea of kconfig entries which may be contrived and could use some
clarifications.

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Greg Kroah-Hartman]

On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > >
> > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > >
> > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > kernels.
> > > > >
> > > > > Christoph, *poke*
> > > >
> > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > 
> > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > are 100% certain we have a direct module --> config mapping.
> > > 
> > > This is *currently* determined when the streamline_config.pl finds
> > > that an object has only *one* associated config symbol associated. As
> > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > system being tested. For the other modules, if they did want this
> > > association, we could allow a way for modules to define their own
> > > KBUILD_KCONF variable so that this could be considered as well, or
> > > they can look at their own kconfig stuff to try to fit the model that
> > > does work. To be clear, the heuristics *can* be updated if there is
> > > confidence in alternative methods for resolution. But since it is
> > > reflective of our current situation, I cannot consider it a hack.
> > > 
> > > This implementation is a reflection of our reality in the kernel, and
> > > as has been discussed in this thread, if we want to correct the gaps
> > > we need to do a lot of work. And *no one* is working towards these
> > > goals.
> > > 
> > > That said, even if you go forward with an intrusive solution like the
> > > one you proposed we could still use the same kconfig_symb...
> > > 
> > > So no, I don't see this as a hack. It's a reflection as to our current
> > > reality. And I cannot see how the kconfig_symb can lie or be
> > > incorrect. So in fact I think that pushing this forward also makes the
> > > problem statement clearer for the future of what semantics needs to be
> > > addressed, and helps us even annotate the problematic areas of the
> > > kernel.
> > > 
> > > What negative aspects do you see with this being merged in practice?
> > 
> > I'm trying to see what the actual problem that you are wanting to solve
> > here with this.  What exactly is it?
> 
> The problem is that there is no current maping of a module to respective
> kconfig symbol.

That's because it is not just "one" symbol per module.

If it were, you can just parse the Makefiles and get that single config
option for most modules, right?  But even then, multiple options can
influence a single module as to what actually gets built into that
module.

So, I would say, "who really cares"?

> > Who needs to determine the
> > "singular" configuration option that caused a kernel module to be built
> > at the expense of all other options?
> 
> Folks wanting to slim down their kernel build, and users of backports.

People who want to "slim" down things are rare, and it's usually worth
it to work backwards anyway (see what functionality is needed and then
go from there, not look at the modules themselves).  Or use a tool like
'make localmodconfig' and trim.

> > What can that be used for and who will use it?
> 
> Without a mapping there is no clean way to let you slim down your kernel
> using a distro kernel as a base, enabling only those things you really
> need.

It's hard to determine "what you really need" :)

Use localmodconfig and you have a great start, then prune from there.
Trying to put _all_ configuration dependencies in a single module isn't
going to work, our configuration language does not distill down to that.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Greg Kroah-Hartman]

On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > >
> > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > >
> > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > kernels.
> > > > >
> > > > > Christoph, *poke*
> > > >
> > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > 
> > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > are 100% certain we have a direct module --> config mapping.
> > > 
> > > This is *currently* determined when the streamline_config.pl finds
> > > that an object has only *one* associated config symbol associated. As
> > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > system being tested. For the other modules, if they did want this
> > > association, we could allow a way for modules to define their own
> > > KBUILD_KCONF variable so that this could be considered as well, or
> > > they can look at their own kconfig stuff to try to fit the model that
> > > does work. To be clear, the heuristics *can* be updated if there is
> > > confidence in alternative methods for resolution. But since it is
> > > reflective of our current situation, I cannot consider it a hack.
> > > 
> > > This implementation is a reflection of our reality in the kernel, and
> > > as has been discussed in this thread, if we want to correct the gaps
> > > we need to do a lot of work. And *no one* is working towards these
> > > goals.
> > > 
> > > That said, even if you go forward with an intrusive solution like the
> > > one you proposed we could still use the same kconfig_symb...
> > > 
> > > So no, I don't see this as a hack. It's a reflection as to our current
> > > reality. And I cannot see how the kconfig_symb can lie or be
> > > incorrect. So in fact I think that pushing this forward also makes the
> > > problem statement clearer for the future of what semantics needs to be
> > > addressed, and helps us even annotate the problematic areas of the
> > > kernel.
> > > 
> > > What negative aspects do you see with this being merged in practice?
> > 
> > I'm trying to see what the actual problem that you are wanting to solve
> > here with this.  What exactly is it?
> 
> The problem is that there is no current maping of a module to respective
> kconfig symbol.

That's because it is not just "one" symbol per module.

If it were, you can just parse the Makefiles and get that single config
option for most modules, right?  But even then, multiple options can
influence a single module as to what actually gets built into that
module.

So, I would say, "who really cares"?

> > Who needs to determine the
> > "singular" configuration option that caused a kernel module to be built
> > at the expense of all other options?
> 
> Folks wanting to slim down their kernel build, and users of backports.

People who want to "slim" down things are rare, and it's usually worth
it to work backwards anyway (see what functionality is needed and then
go from there, not look at the modules themselves).  Or use a tool like
'make localmodconfig' and trim.

> > What can that be used for and who will use it?
> 
> Without a mapping there is no clean way to let you slim down your kernel
> using a distro kernel as a base, enabling only those things you really
> need.

It's hard to determine "what you really need" :)

Use localmodconfig and you have a great start, then prune from there.
Trying to put _all_ configuration dependencies in a single module isn't
going to work, our configuration language does not distill down to that.

thanks,

greg k-h

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > >
> > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > >
> > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > kernels.
> > > > > >
> > > > > > Christoph, *poke*
> > > > >
> > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > 
> > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > are 100% certain we have a direct module --> config mapping.
> > > > 
> > > > This is *currently* determined when the streamline_config.pl finds
> > > > that an object has only *one* associated config symbol associated. As
> > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > system being tested. For the other modules, if they did want this
> > > > association, we could allow a way for modules to define their own
> > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > they can look at their own kconfig stuff to try to fit the model that
> > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > confidence in alternative methods for resolution. But since it is
> > > > reflective of our current situation, I cannot consider it a hack.
> > > > 
> > > > This implementation is a reflection of our reality in the kernel, and
> > > > as has been discussed in this thread, if we want to correct the gaps
> > > > we need to do a lot of work. And *no one* is working towards these
> > > > goals.
> > > > 
> > > > That said, even if you go forward with an intrusive solution like the
> > > > one you proposed we could still use the same kconfig_symb...
> > > > 
> > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > problem statement clearer for the future of what semantics needs to be
> > > > addressed, and helps us even annotate the problematic areas of the
> > > > kernel.
> > > > 
> > > > What negative aspects do you see with this being merged in practice?
> > > 
> > > I'm trying to see what the actual problem that you are wanting to solve
> > > here with this.  What exactly is it?
> > 
> > The problem is that there is no current maping of a module to respective
> > kconfig symbol.
> 
> That's because it is not just "one" symbol per module.

This is true. But it is not the case for all modules.  In fact it seems
its true that most modules do have *one* main symbol.

On at least Cristina's system of of 62 modules 58 *did* have one symbol.
For the modules evaluated where this was not the case, it did seem wise
to actually revise the symbol definition for the other modules.

> If it were, you can just parse the Makefiles and get that single config
> option for most modules, right?

The heuristic essentially does this and only provides the module
attribute where this was true.

> But even then, multiple options can
> influence a single module as to what actually gets built into that
> module.

Yes. For example one hardware device driver may support different
families of chipsets, and so it could have sub-options for each
family.

> So, I would say, "who really cares"?

For most visible modules it would seem we do have a one config symbol
mapping which could enable it. And I noted who would care. The defaults
of a module, for instance sub-options to enable / disable different
support for different chipsets, *should* suffice for most users.

> > > Who needs to determine the
> > > "singular" configuration option that caused a kernel module to be built
> > > at the expense of all other options?
> > 
> > Folks wanting to slim down their kernel build, and users of backports.
> 
> People who want to "slim" down things are rare, 

It is basic math though:

Users of 'make localmodconfig' +  backport users > Users of 'make localmodconfig'

And yet we already support 'make localmodconfig'. So what is being
proposed can help enhance 'make localmodconfig' and yet provides more
users outside of those users, without requiring kernel sources.

> and it's usually worth
> it to work backwards anyway (see what functionality is needed and then
> go from there, not look at the modules themselves).  Or use a tool like
> 'make localmodconfig' and trim.

And I am noting we can further enhance a feature which we already
do support, and enable *more* users requiring similar information.

> > > What can that be used for and who will use it?
> > 
> > Without a mapping there is no clean way to let you slim down your kernel
> > using a distro kernel as a base, enabling only those things you really
> > need.
> 
> It's hard to determine "what you really need" :)

Right, but at least for device functionality, the above would help
significantly. It also poses the question whether or not device drivers
*should* strive towards having one kconfig symbol to help with this.

There is a lot of research over the lack of proper semantics on use of
kconfig, and issues this causes. It is so bad that some researchers have
tried define our semantics through *reverse engineering*. The question
of whether or not we *should* strive to have *one* symbol per a driver
for an actual hardware component is worth evaluating long term, for the
sake of helping with semantics of kconfig use. I see there being gains
with this, and I find it hard to find counters to where quite the
opposite is true. Can you?

> Use localmodconfig and you have a great start, then prune from there.

This thread poses the question if we can do better, and suggests one
small area where we can start.

> Trying to put _all_ configuration dependencies in a single module isn't
> going to work, our configuration language does not distill down to that.

The question we should be evaluating if we *should* strive to buckle up
on this and have at least one config symbol per module associated with
hardware. I'm suggesting there are gains for this, and am providing two
groups of users that would benefit from this clearly. And I'm also
suggesting that it could help with kconfig semantics, long term.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > >
> > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > >
> > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > kernels.
> > > > > >
> > > > > > Christoph, *poke*
> > > > >
> > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > 
> > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > are 100% certain we have a direct module --> config mapping.
> > > > 
> > > > This is *currently* determined when the streamline_config.pl finds
> > > > that an object has only *one* associated config symbol associated. As
> > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > system being tested. For the other modules, if they did want this
> > > > association, we could allow a way for modules to define their own
> > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > they can look at their own kconfig stuff to try to fit the model that
> > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > confidence in alternative methods for resolution. But since it is
> > > > reflective of our current situation, I cannot consider it a hack.
> > > > 
> > > > This implementation is a reflection of our reality in the kernel, and
> > > > as has been discussed in this thread, if we want to correct the gaps
> > > > we need to do a lot of work. And *no one* is working towards these
> > > > goals.
> > > > 
> > > > That said, even if you go forward with an intrusive solution like the
> > > > one you proposed we could still use the same kconfig_symb...
> > > > 
> > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > problem statement clearer for the future of what semantics needs to be
> > > > addressed, and helps us even annotate the problematic areas of the
> > > > kernel.
> > > > 
> > > > What negative aspects do you see with this being merged in practice?
> > > 
> > > I'm trying to see what the actual problem that you are wanting to solve
> > > here with this.  What exactly is it?
> > 
> > The problem is that there is no current maping of a module to respective
> > kconfig symbol.
> 
> That's because it is not just "one" symbol per module.

This is true. But it is not the case for all modules.  In fact it seems
its true that most modules do have *one* main symbol.

On at least Cristina's system of of 62 modules 58 *did* have one symbol.
For the modules evaluated where this was not the case, it did seem wise
to actually revise the symbol definition for the other modules.

> If it were, you can just parse the Makefiles and get that single config
> option for most modules, right?

The heuristic essentially does this and only provides the module
attribute where this was true.

> But even then, multiple options can
> influence a single module as to what actually gets built into that
> module.

Yes. For example one hardware device driver may support different
families of chipsets, and so it could have sub-options for each
family.

> So, I would say, "who really cares"?

For most visible modules it would seem we do have a one config symbol
mapping which could enable it. And I noted who would care. The defaults
of a module, for instance sub-options to enable / disable different
support for different chipsets, *should* suffice for most users.

> > > Who needs to determine the
> > > "singular" configuration option that caused a kernel module to be built
> > > at the expense of all other options?
> > 
> > Folks wanting to slim down their kernel build, and users of backports.
> 
> People who want to "slim" down things are rare, 

It is basic math though:

Users of 'make localmodconfig' +  backport users > Users of 'make localmodconfig'

And yet we already support 'make localmodconfig'. So what is being
proposed can help enhance 'make localmodconfig' and yet provides more
users outside of those users, without requiring kernel sources.

> and it's usually worth
> it to work backwards anyway (see what functionality is needed and then
> go from there, not look at the modules themselves).  Or use a tool like
> 'make localmodconfig' and trim.

And I am noting we can further enhance a feature which we already
do support, and enable *more* users requiring similar information.

> > > What can that be used for and who will use it?
> > 
> > Without a mapping there is no clean way to let you slim down your kernel
> > using a distro kernel as a base, enabling only those things you really
> > need.
> 
> It's hard to determine "what you really need" :)

Right, but at least for device functionality, the above would help
significantly. It also poses the question whether or not device drivers
*should* strive towards having one kconfig symbol to help with this.

There is a lot of research over the lack of proper semantics on use of
kconfig, and issues this causes. It is so bad that some researchers have
tried define our semantics through *reverse engineering*. The question
of whether or not we *should* strive to have *one* symbol per a driver
for an actual hardware component is worth evaluating long term, for the
sake of helping with semantics of kconfig use. I see there being gains
with this, and I find it hard to find counters to where quite the
opposite is true. Can you?

> Use localmodconfig and you have a great start, then prune from there.

This thread poses the question if we can do better, and suggests one
small area where we can start.

> Trying to put _all_ configuration dependencies in a single module isn't
> going to work, our configuration language does not distill down to that.

The question we should be evaluating if we *should* strive to buckle up
on this and have at least one config symbol per module associated with
hardware. I'm suggesting there are gains for this, and am providing two
groups of users that would benefit from this clearly. And I'm also
suggesting that it could help with kconfig semantics, long term.

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Greg Kroah-Hartman]

On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > > >
> > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > >
> > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > kernels.
> > > > > > >
> > > > > > > Christoph, *poke*
> > > > > >
> > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > 
> > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > are 100% certain we have a direct module --> config mapping.
> > > > > 
> > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > that an object has only *one* associated config symbol associated. As
> > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > system being tested. For the other modules, if they did want this
> > > > > association, we could allow a way for modules to define their own
> > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > confidence in alternative methods for resolution. But since it is
> > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > 
> > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > goals.
> > > > > 
> > > > > That said, even if you go forward with an intrusive solution like the
> > > > > one you proposed we could still use the same kconfig_symb...
> > > > > 
> > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > problem statement clearer for the future of what semantics needs to be
> > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > kernel.
> > > > > 
> > > > > What negative aspects do you see with this being merged in practice?
> > > > 
> > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > here with this.  What exactly is it?
> > > 
> > > The problem is that there is no current maping of a module to respective
> > > kconfig symbol.
> > 
> > That's because it is not just "one" symbol per module.
> 
> This is true. But it is not the case for all modules.  In fact it seems
> its true that most modules do have *one* main symbol.

You mean "one unique symbol from all other modules", right?

That is much different than just "one" symbol, given that almost
every driver depends on something else being enabled as well (bus type,
platform type, arch, etc.)

And I would argue, that finding that "one" symbol is easy, just parse
the Makefiles.  But I would also state that this "one" symbol doesn't
really help you much as those are the "simple" things.  It's how to
turn on all of the required symbols to get to that "one" symbol that is
the hard part.

And conversely, if you disable that "one" symbol, does that also mean
you can disable the symbols it depended on?  If so, how far back?

And what about functionality?  If my usb-storage device is "enabled" in
the build, yet all filesystems are not, or the needed dm module is not,
it is useless.  Hardware requires usually more than one real "symbol" in
order to work properly, as you know.

And of course, what does this really matter to anyone?  If you build
"all modules" and you only load the modules you actually use for your
hardware (based on auto-loading), then your system uses the same amount
of memory as if you disabled all of the modules you did not need.

Yes, it's faster to build, but is that what you are trying to optimize
for here?

Anyway, if this is just an acidemic thing, have fun, but I would not be
adding anything else to the module image that is not really going to be
useful to anyone.

good luck!

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Greg Kroah-Hartman]

On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > > >
> > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > >
> > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > kernels.
> > > > > > >
> > > > > > > Christoph, *poke*
> > > > > >
> > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > 
> > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > are 100% certain we have a direct module --> config mapping.
> > > > > 
> > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > that an object has only *one* associated config symbol associated. As
> > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > system being tested. For the other modules, if they did want this
> > > > > association, we could allow a way for modules to define their own
> > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > confidence in alternative methods for resolution. But since it is
> > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > 
> > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > goals.
> > > > > 
> > > > > That said, even if you go forward with an intrusive solution like the
> > > > > one you proposed we could still use the same kconfig_symb...
> > > > > 
> > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > problem statement clearer for the future of what semantics needs to be
> > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > kernel.
> > > > > 
> > > > > What negative aspects do you see with this being merged in practice?
> > > > 
> > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > here with this.  What exactly is it?
> > > 
> > > The problem is that there is no current maping of a module to respective
> > > kconfig symbol.
> > 
> > That's because it is not just "one" symbol per module.
> 
> This is true. But it is not the case for all modules.  In fact it seems
> its true that most modules do have *one* main symbol.

You mean "one unique symbol from all other modules", right?

That is much different than just "one" symbol, given that almost
every driver depends on something else being enabled as well (bus type,
platform type, arch, etc.)

And I would argue, that finding that "one" symbol is easy, just parse
the Makefiles.  But I would also state that this "one" symbol doesn't
really help you much as those are the "simple" things.  It's how to
turn on all of the required symbols to get to that "one" symbol that is
the hard part.

And conversely, if you disable that "one" symbol, does that also mean
you can disable the symbols it depended on?  If so, how far back?

And what about functionality?  If my usb-storage device is "enabled" in
the build, yet all filesystems are not, or the needed dm module is not,
it is useless.  Hardware requires usually more than one real "symbol" in
order to work properly, as you know.

And of course, what does this really matter to anyone?  If you build
"all modules" and you only load the modules you actually use for your
hardware (based on auto-loading), then your system uses the same amount
of memory as if you disabled all of the modules you did not need.

Yes, it's faster to build, but is that what you are trying to optimize
for here?

Anyway, if this is just an acidemic thing, have fun, but I would not be
adding anything else to the module image that is not really going to be
useful to anyone.

good luck!

greg k-h

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> > On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > > >
> > > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > > kernels.
> > > > > > > >
> > > > > > > > Christoph, *poke*
> > > > > > >
> > > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > > 
> > > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > > are 100% certain we have a direct module --> config mapping.
> > > > > > 
> > > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > > that an object has only *one* associated config symbol associated. As
> > > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > > system being tested. For the other modules, if they did want this
> > > > > > association, we could allow a way for modules to define their own
> > > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > > confidence in alternative methods for resolution. But since it is
> > > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > > 
> > > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > > goals.
> > > > > > 
> > > > > > That said, even if you go forward with an intrusive solution like the
> > > > > > one you proposed we could still use the same kconfig_symb...
> > > > > > 
> > > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > > problem statement clearer for the future of what semantics needs to be
> > > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > > kernel.
> > > > > > 
> > > > > > What negative aspects do you see with this being merged in practice?
> > > > > 
> > > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > > here with this.  What exactly is it?
> > > > 
> > > > The problem is that there is no current maping of a module to respective
> > > > kconfig symbol.
> > > 
> > > That's because it is not just "one" symbol per module.
> > 
> > This is true. But it is not the case for all modules.  In fact it seems
> > its true that most modules do have *one* main symbol.
> 
> You mean "one unique symbol from all other modules", right?

I mean that in the typical case you have *one* main *parent* symbol defining
one device driver for one piece of hardware.

> That is much different than just "one" symbol, given that almost
> every driver depends on something else being enabled as well (bus type,
> platform type, arch, etc.)

Yes but that is different than what is trying to be addressed.

> And I would argue, that finding that "one" symbol is easy, just parse
> the Makefiles.  But I would also state that this "one" symbol doesn't
> really help you much as those are the "simple" things.  It's how to
> turn on all of the required symbols to get to that "one" symbol that is
> the hard part.

Absolutely, but *that* is a different problem! But I am glad you brought
that up and acknowledge it. Addressing that problem will take time and
folks are working towards it. Addressing that problem will still however
not address the problem being addressed here.

> And conversely, if you disable that "one" symbol, does that also mean
> you can disable the symbols it depended on?  If so, how far back?

Right.. 

> And what about functionality?  If my usb-storage device is "enabled" in
> the build, yet all filesystems are not, or the needed dm module is not,
> it is useless.

Yes. The localmodconfig approach is to keep enabled as modules *all* currently
enabled modules, that covers that.

But again, this is a separate problem. The one I am addressing, on
behalf of Cristina, is a subspace, dedicated towards *hardware*
functionality.

> Hardware requires usually more than one real "symbol" in
> order to work properly, as you know.

Right. This does not mean that this information of parent main symbol is
not useful. And having it available on the modules can help with
multiple goals, without requiring kernel sources.

> And of course, what does this really matter to anyone?  If you build
> "all modules" and you only load the modules you actually use for your
> hardware (based on auto-loading), then your system uses the same amount
> of memory as if you disabled all of the modules you did not need.

For backports it means you can enable only what you need, or at least
show users what modules in a backports release could be upgraded and let
you enable / disable them.  For other users, people can care about build
time. And not everyone has fancy systems to build all modules; and
sometimes you may just want to enable a small qemu system and build
locally on it, enabling all modules on such systems would just be
extremely silly.

> Yes, it's faster to build, but is that what you are trying to optimize
> for here?

Particularly for me, yes. Others have other needs and I have already
stated clearly what the gains are.

> Anyway, if this is just an acidemic thing, have fun, but I would not be
> adding anything else to the module image that is not really going to be
> useful to anyone.

The academic consensus is kconfig semantics are poo and we need to do
slowly start addressing this. I believe that striving towards having
one kconfig symbol per hw component can help long term, and in the
meantime, this also will help with the 'make localmodconfig' and
backport users.

> good luck!

This is not about luck. If you don't want to address these problems,
that's fine, but please provide objective considerations rather than
right out rejection without a reasonable basis. Or even better, provide
alternative recommendations.

The alternative Christoph recommended is hugely instrusive, no one is
working on it, and the simple solution proposed in this thread at least
gets us a small step forward in helping to enable a few more users,
while also postulating if we should strive towards having one main kconfig
for each hw component. Since it seems this is already the case, and
there are only a few outliers, this effort should help spot outliers
and address them to help with our semantics.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> > On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > > >
> > > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > > kernels.
> > > > > > > >
> > > > > > > > Christoph, *poke*
> > > > > > >
> > > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > > 
> > > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > > are 100% certain we have a direct module --> config mapping.
> > > > > > 
> > > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > > that an object has only *one* associated config symbol associated. As
> > > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > > system being tested. For the other modules, if they did want this
> > > > > > association, we could allow a way for modules to define their own
> > > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > > confidence in alternative methods for resolution. But since it is
> > > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > > 
> > > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > > goals.
> > > > > > 
> > > > > > That said, even if you go forward with an intrusive solution like the
> > > > > > one you proposed we could still use the same kconfig_symb...
> > > > > > 
> > > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > > problem statement clearer for the future of what semantics needs to be
> > > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > > kernel.
> > > > > > 
> > > > > > What negative aspects do you see with this being merged in practice?
> > > > > 
> > > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > > here with this.  What exactly is it?
> > > > 
> > > > The problem is that there is no current maping of a module to respective
> > > > kconfig symbol.
> > > 
> > > That's because it is not just "one" symbol per module.
> > 
> > This is true. But it is not the case for all modules.  In fact it seems
> > its true that most modules do have *one* main symbol.
> 
> You mean "one unique symbol from all other modules", right?

I mean that in the typical case you have *one* main *parent* symbol defining
one device driver for one piece of hardware.

> That is much different than just "one" symbol, given that almost
> every driver depends on something else being enabled as well (bus type,
> platform type, arch, etc.)

Yes but that is different than what is trying to be addressed.

> And I would argue, that finding that "one" symbol is easy, just parse
> the Makefiles.  But I would also state that this "one" symbol doesn't
> really help you much as those are the "simple" things.  It's how to
> turn on all of the required symbols to get to that "one" symbol that is
> the hard part.

Absolutely, but *that* is a different problem! But I am glad you brought
that up and acknowledge it. Addressing that problem will take time and
folks are working towards it. Addressing that problem will still however
not address the problem being addressed here.

> And conversely, if you disable that "one" symbol, does that also mean
> you can disable the symbols it depended on?  If so, how far back?

Right.. 

> And what about functionality?  If my usb-storage device is "enabled" in
> the build, yet all filesystems are not, or the needed dm module is not,
> it is useless.

Yes. The localmodconfig approach is to keep enabled as modules *all* currently
enabled modules, that covers that.

But again, this is a separate problem. The one I am addressing, on
behalf of Cristina, is a subspace, dedicated towards *hardware*
functionality.

> Hardware requires usually more than one real "symbol" in
> order to work properly, as you know.

Right. This does not mean that this information of parent main symbol is
not useful. And having it available on the modules can help with
multiple goals, without requiring kernel sources.

> And of course, what does this really matter to anyone?  If you build
> "all modules" and you only load the modules you actually use for your
> hardware (based on auto-loading), then your system uses the same amount
> of memory as if you disabled all of the modules you did not need.

For backports it means you can enable only what you need, or at least
show users what modules in a backports release could be upgraded and let
you enable / disable them.  For other users, people can care about build
time. And not everyone has fancy systems to build all modules; and
sometimes you may just want to enable a small qemu system and build
locally on it, enabling all modules on such systems would just be
extremely silly.

> Yes, it's faster to build, but is that what you are trying to optimize
> for here?

Particularly for me, yes. Others have other needs and I have already
stated clearly what the gains are.

> Anyway, if this is just an acidemic thing, have fun, but I would not be
> adding anything else to the module image that is not really going to be
> useful to anyone.

The academic consensus is kconfig semantics are poo and we need to do
slowly start addressing this. I believe that striving towards having
one kconfig symbol per hw component can help long term, and in the
meantime, this also will help with the 'make localmodconfig' and
backport users.

> good luck!

This is not about luck. If you don't want to address these problems,
that's fine, but please provide objective considerations rather than
right out rejection without a reasonable basis. Or even better, provide
alternative recommendations.

The alternative Christoph recommended is hugely instrusive, no one is
working on it, and the simple solution proposed in this thread at least
gets us a small step forward in helping to enable a few more users,
while also postulating if we should strive towards having one main kconfig
for each hw component. Since it seems this is already the case, and
there are only a few outliers, this effort should help spot outliers
and address them to help with our semantics.

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Brendan Higgins]

On Wed, Jul 3, 2019 at 3:25 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> > On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> > > On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > > > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > > > >
> > > > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > > > kernels.
> > > > > > > > >
> > > > > > > > > Christoph, *poke*
> > > > > > > >
> > > > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > > >
> > > > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > > > are 100% certain we have a direct module --> config mapping.
> > > > > > >
> > > > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > > > that an object has only *one* associated config symbol associated. As
> > > > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > > > system being tested. For the other modules, if they did want this
> > > > > > > association, we could allow a way for modules to define their own
> > > > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > > > confidence in alternative methods for resolution. But since it is
> > > > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > > >
> > > > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > > > goals.
> > > > > > >
> > > > > > > That said, even if you go forward with an intrusive solution like the
> > > > > > > one you proposed we could still use the same kconfig_symb...
> > > > > > >
> > > > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > > > problem statement clearer for the future of what semantics needs to be
> > > > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > > > kernel.
> > > > > > >
> > > > > > > What negative aspects do you see with this being merged in practice?
> > > > > >
> > > > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > > > here with this.  What exactly is it?
> > > > >
> > > > > The problem is that there is no current maping of a module to respective
> > > > > kconfig symbol.
> > > >
> > > > That's because it is not just "one" symbol per module.
> > >
> > > This is true. But it is not the case for all modules.  In fact it seems
> > > its true that most modules do have *one* main symbol.
> >
> > You mean "one unique symbol from all other modules", right?
>
> I mean that in the typical case you have *one* main *parent* symbol defining
> one device driver for one piece of hardware.
>
> > That is much different than just "one" symbol, given that almost
> > every driver depends on something else being enabled as well (bus type,
> > platform type, arch, etc.)
>
> Yes but that is different than what is trying to be addressed.
>
> > And I would argue, that finding that "one" symbol is easy, just parse
> > the Makefiles.  But I would also state that this "one" symbol doesn't
> > really help you much as those are the "simple" things.  It's how to
> > turn on all of the required symbols to get to that "one" symbol that is
> > the hard part.
>
> Absolutely, but *that* is a different problem! But I am glad you brought
> that up and acknowledge it. Addressing that problem will take time and
> folks are working towards it. Addressing that problem will still however
> not address the problem being addressed here.
>
> > And conversely, if you disable that "one" symbol, does that also mean
> > you can disable the symbols it depended on?  If so, how far back?
>
> Right..
>
> > And what about functionality?  If my usb-storage device is "enabled" in
> > the build, yet all filesystems are not, or the needed dm module is not,
> > it is useless.
>
> Yes. The localmodconfig approach is to keep enabled as modules *all* currently
> enabled modules, that covers that.
>
> But again, this is a separate problem. The one I am addressing, on
> behalf of Cristina, is a subspace, dedicated towards *hardware*
> functionality.

Hmm...maybe this isn't what I am looking for then. I am interested in
the problem of figuring out what dependencies I need to select to turn
on a desired config symbol (which is obviously a separate issue), and
I am interested in associating symbols with a config symbol and then
ensuring that symbol is exercised. Basically, I want a way to make
sure my tests actually get run without a human looking at them; I feel
like what you are working on might help with this latter issue, but I
am not 100% sure. It sounds like it is not your primary goal in any
case.

> > Hardware requires usually more than one real "symbol" in
> > order to work properly, as you know.
>
> Right. This does not mean that this information of parent main symbol is
> not useful. And having it available on the modules can help with
> multiple goals, without requiring kernel sources.
>
> > And of course, what does this really matter to anyone?  If you build
> > "all modules" and you only load the modules you actually use for your
> > hardware (based on auto-loading), then your system uses the same amount
> > of memory as if you disabled all of the modules you did not need.
>
> For backports it means you can enable only what you need, or at least
> show users what modules in a backports release could be upgraded and let
> you enable / disable them.  For other users, people can care about build
> time. And not everyone has fancy systems to build all modules; and
> sometimes you may just want to enable a small qemu system and build
> locally on it, enabling all modules on such systems would just be
> extremely silly.
>
> > Yes, it's faster to build, but is that what you are trying to optimize
> > for here?
>
> Particularly for me, yes. Others have other needs and I have already
> stated clearly what the gains are.
>
> > Anyway, if this is just an acidemic thing, have fun, but I would not be
> > adding anything else to the module image that is not really going to be
> > useful to anyone.
>
> The academic consensus is kconfig semantics are poo and we need to do
> slowly start addressing this. I believe that striving towards having
> one kconfig symbol per hw component can help long term, and in the
> meantime, this also will help with the 'make localmodconfig' and
> backport users.
>
> > good luck!
>
> This is not about luck. If you don't want to address these problems,
> that's fine, but please provide objective considerations rather than
> right out rejection without a reasonable basis. Or even better, provide
> alternative recommendations.
>
> The alternative Christoph recommended is hugely instrusive, no one is
> working on it, and the simple solution proposed in this thread at least
> gets us a small step forward in helping to enable a few more users,
> while also postulating if we should strive towards having one main kconfig
> for each hw component. Since it seems this is already the case, and
> there are only a few outliers, this effort should help spot outliers
> and address them to help with our semantics.

Ooo, looks like I should take a look at that.

Cheers!
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Brendan Higgins]

On Wed, Jul 3, 2019 at 3:25 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> > On Wed, Jul 03, 2019 at 04:50:20PM +0000, Luis Chamberlain wrote:
> > > On Wed, Jul 03, 2019 at 09:40:48AM +0200, Greg Kroah-Hartman wrote:
> > > > On Tue, Jul 02, 2019 at 08:51:06PM +0000, Luis Chamberlain wrote:
> > > > > On Sat, Jun 29, 2019 at 10:42:57AM +0200, Greg Kroah-Hartman wrote:
> > > > > > On Fri, Jun 28, 2019 at 11:40:22AM -0700, Luis Chamberlain wrote:
> > > > > > > On Wed, Jun 26, 2019 at 9:51 PM Christoph Hellwig <hch@lst.de> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jun 26, 2019 at 03:21:08PM -0700, Luis Chamberlain wrote:
> > > > > > > > > On Tue, Feb 5, 2019 at 2:07 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > > > > > > > > In lieu of no Luke Skywalker, if you will, for a large kconfig revamp
> > > > > > > > > > on this, I'm inclined to believe *at least* having some kconfig_symb
> > > > > > > > > > exposed for some modules is better than nothing. Christoph are you
> > > > > > > > > > totally opposed to this effort until we get a non-reverse engineered
> > > > > > > > > > effort in place? It just seems like an extraordinary amount of work
> > > > > > > > > > and I'm not quite sure who's volunteering to do it.
> > > > > > > > > >
> > > > > > > > > > Other stakeholders may benefit from at least having some config -->
> > > > > > > > > > module mapping for now. Not just backports or building slimmer
> > > > > > > > > > kernels.
> > > > > > > > >
> > > > > > > > > Christoph, *poke*
> > > > > > > >
> > > > > > > > Yes, I'm still totally opposed to a half-backed hack like this.
> > > > > > >
> > > > > > > The solution puts forward a mechanism to add a kconfig_symb where we
> > > > > > > are 100% certain we have a direct module --> config mapping.
> > > > > > >
> > > > > > > This is *currently* determined when the streamline_config.pl finds
> > > > > > > that an object has only *one* associated config symbol associated. As
> > > > > > > Cristina noted, of 62 modules on a running system 58 of them ended up
> > > > > > > getting the kconfig_symb assigned, that is 93.5% of all modules on the
> > > > > > > system being tested. For the other modules, if they did want this
> > > > > > > association, we could allow a way for modules to define their own
> > > > > > > KBUILD_KCONF variable so that this could be considered as well, or
> > > > > > > they can look at their own kconfig stuff to try to fit the model that
> > > > > > > does work. To be clear, the heuristics *can* be updated if there is
> > > > > > > confidence in alternative methods for resolution. But since it is
> > > > > > > reflective of our current situation, I cannot consider it a hack.
> > > > > > >
> > > > > > > This implementation is a reflection of our reality in the kernel, and
> > > > > > > as has been discussed in this thread, if we want to correct the gaps
> > > > > > > we need to do a lot of work. And *no one* is working towards these
> > > > > > > goals.
> > > > > > >
> > > > > > > That said, even if you go forward with an intrusive solution like the
> > > > > > > one you proposed we could still use the same kconfig_symb...
> > > > > > >
> > > > > > > So no, I don't see this as a hack. It's a reflection as to our current
> > > > > > > reality. And I cannot see how the kconfig_symb can lie or be
> > > > > > > incorrect. So in fact I think that pushing this forward also makes the
> > > > > > > problem statement clearer for the future of what semantics needs to be
> > > > > > > addressed, and helps us even annotate the problematic areas of the
> > > > > > > kernel.
> > > > > > >
> > > > > > > What negative aspects do you see with this being merged in practice?
> > > > > >
> > > > > > I'm trying to see what the actual problem that you are wanting to solve
> > > > > > here with this.  What exactly is it?
> > > > >
> > > > > The problem is that there is no current maping of a module to respective
> > > > > kconfig symbol.
> > > >
> > > > That's because it is not just "one" symbol per module.
> > >
> > > This is true. But it is not the case for all modules.  In fact it seems
> > > its true that most modules do have *one* main symbol.
> >
> > You mean "one unique symbol from all other modules", right?
>
> I mean that in the typical case you have *one* main *parent* symbol defining
> one device driver for one piece of hardware.
>
> > That is much different than just "one" symbol, given that almost
> > every driver depends on something else being enabled as well (bus type,
> > platform type, arch, etc.)
>
> Yes but that is different than what is trying to be addressed.
>
> > And I would argue, that finding that "one" symbol is easy, just parse
> > the Makefiles.  But I would also state that this "one" symbol doesn't
> > really help you much as those are the "simple" things.  It's how to
> > turn on all of the required symbols to get to that "one" symbol that is
> > the hard part.
>
> Absolutely, but *that* is a different problem! But I am glad you brought
> that up and acknowledge it. Addressing that problem will take time and
> folks are working towards it. Addressing that problem will still however
> not address the problem being addressed here.
>
> > And conversely, if you disable that "one" symbol, does that also mean
> > you can disable the symbols it depended on?  If so, how far back?
>
> Right..
>
> > And what about functionality?  If my usb-storage device is "enabled" in
> > the build, yet all filesystems are not, or the needed dm module is not,
> > it is useless.
>
> Yes. The localmodconfig approach is to keep enabled as modules *all* currently
> enabled modules, that covers that.
>
> But again, this is a separate problem. The one I am addressing, on
> behalf of Cristina, is a subspace, dedicated towards *hardware*
> functionality.

Hmm...maybe this isn't what I am looking for then. I am interested in
the problem of figuring out what dependencies I need to select to turn
on a desired config symbol (which is obviously a separate issue), and
I am interested in associating symbols with a config symbol and then
ensuring that symbol is exercised. Basically, I want a way to make
sure my tests actually get run without a human looking at them; I feel
like what you are working on might help with this latter issue, but I
am not 100% sure. It sounds like it is not your primary goal in any
case.

> > Hardware requires usually more than one real "symbol" in
> > order to work properly, as you know.
>
> Right. This does not mean that this information of parent main symbol is
> not useful. And having it available on the modules can help with
> multiple goals, without requiring kernel sources.
>
> > And of course, what does this really matter to anyone?  If you build
> > "all modules" and you only load the modules you actually use for your
> > hardware (based on auto-loading), then your system uses the same amount
> > of memory as if you disabled all of the modules you did not need.
>
> For backports it means you can enable only what you need, or at least
> show users what modules in a backports release could be upgraded and let
> you enable / disable them.  For other users, people can care about build
> time. And not everyone has fancy systems to build all modules; and
> sometimes you may just want to enable a small qemu system and build
> locally on it, enabling all modules on such systems would just be
> extremely silly.
>
> > Yes, it's faster to build, but is that what you are trying to optimize
> > for here?
>
> Particularly for me, yes. Others have other needs and I have already
> stated clearly what the gains are.
>
> > Anyway, if this is just an acidemic thing, have fun, but I would not be
> > adding anything else to the module image that is not really going to be
> > useful to anyone.
>
> The academic consensus is kconfig semantics are poo and we need to do
> slowly start addressing this. I believe that striving towards having
> one kconfig symbol per hw component can help long term, and in the
> meantime, this also will help with the 'make localmodconfig' and
> backport users.
>
> > good luck!
>
> This is not about luck. If you don't want to address these problems,
> that's fine, but please provide objective considerations rather than
> right out rejection without a reasonable basis. Or even better, provide
> alternative recommendations.
>
> The alternative Christoph recommended is hugely instrusive, no one is
> working on it, and the simple solution proposed in this thread at least
> gets us a small step forward in helping to enable a few more users,
> while also postulating if we should strive towards having one main kconfig
> for each hw component. Since it seems this is already the case, and
> there are only a few outliers, this effort should help spot outliers
> and address them to help with our semantics.

Ooo, looks like I should take a look at that.

Cheers!

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Thu, Jul 11, 2019 at 04:07:27PM -0700, Brendan Higgins wrote:
> On Wed, Jul 3, 2019 at 3:25 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> >
> > But again, this is a separate problem. The one I am addressing, on
> > behalf of Cristina, is a subspace, dedicated towards *hardware*
> > functionality.
> 
> Hmm...maybe this isn't what I am looking for then. I am interested in
> the problem of figuring out what dependencies I need to select to turn
> on a desired config symbol (which is obviously a separate issue),

It isn't directly. the problem statement is different, and
for that I do recommend checking out the kconfig-sat effort.

It only indirectly relates in that we already know our kconfig
semantics need work, and I do think that the underlying problem
in this thread slowly strives towards addressing kconfig semantics
for modules, associating one main kconfig symbol with one module.

I have no evidence for this though. But since there are users who can
gain from it, I don't see any issues from embracing it.

> and
> I am interested in associating symbols with a config symbol and then
> ensuring that symbol is exercised. Basically, I want a way to make
> sure my tests actually get run without a human looking at them; I feel
> like what you are working on might help with this latter issue, but I
> am not 100% sure. It sounds like it is not your primary goal in any
> case.

Nope, this topic only ralates to yours in the semantics case.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Thu, Jul 11, 2019 at 04:07:27PM -0700, Brendan Higgins wrote:
> On Wed, Jul 3, 2019 at 3:25 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > On Wed, Jul 03, 2019 at 08:57:58PM +0200, Greg Kroah-Hartman wrote:
> >
> > But again, this is a separate problem. The one I am addressing, on
> > behalf of Cristina, is a subspace, dedicated towards *hardware*
> > functionality.
> 
> Hmm...maybe this isn't what I am looking for then. I am interested in
> the problem of figuring out what dependencies I need to select to turn
> on a desired config symbol (which is obviously a separate issue),

It isn't directly. the problem statement is different, and
for that I do recommend checking out the kconfig-sat effort.

It only indirectly relates in that we already know our kconfig
semantics need work, and I do think that the underlying problem
in this thread slowly strives towards addressing kconfig semantics
for modules, associating one main kconfig symbol with one module.

I have no evidence for this though. But since there are users who can
gain from it, I don't see any issues from embracing it.

> and
> I am interested in associating symbols with a config symbol and then
> ensuring that symbol is exercised. Basically, I want a way to make
> sure my tests actually get run without a human looking at them; I feel
> like what you are working on might help with this latter issue, but I
> am not 100% sure. It sounds like it is not your primary goal in any
> case.

Nope, this topic only ralates to yours in the semantics case.

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Enrico Weigelt, metux IT consult]

On 28.06.19 20:40, Luis Chamberlain wrote:

Hi folks,

> The solution puts forward a mechanism to add a kconfig_symb where we> are 100% certain we have a direct module --> config mapping.
Okay, but IIRC this will add more boilerplate those modules.

And I wonder whether target binaries are the right place for those
things at all - IMHO that's something one wants to derive from the
source code  / .config's. At least in the cases I'm imagining, I don't
even have an actual kernel running on the corresponding target yet.
(eg. in crosscompile situations)

OTOH, a more pressing problem for me is identifying the right drivers
and corresponding config options (usually plural, as certain subsystems
have to be enabled, too) by hardware information like DT, ACPI, DMI,
PCI, etc. For now, I have to do that manually, which is pretty time
consuming.

In embedded world, we often have scenarios where we want a really
minimal kernel, but need to enable/disable certain hi-level peripherals
in the middle of the project (eg. "oh, we also need ethernet, but we
wanna drop usb"). There we'll have to find out what actual chip is,
its corresponding driver, required subsystems, etc, and also kick off
everything we don't need anymore.

I've thought about implementing some actual dependency tracking
(at least recording the auto-enabled symbols), but didn't expect that
to become practically usable anytime soon, so I went for a different
approach: writing a little tool that allows modeling hilevel features
and corresponding (potentially board-specific) config syms, so the whole
.config for certain board and usecase can be autogenerated by just some
small meta-configuration:

   https://github.com/metux/kmct

Maybe this could also help for your usecase ?


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Enrico Weigelt, metux IT consult]

On 28.06.19 20:40, Luis Chamberlain wrote:

Hi folks,

> The solution puts forward a mechanism to add a kconfig_symb where we> are 100% certain we have a direct module --> config mapping.
Okay, but IIRC this will add more boilerplate those modules.

And I wonder whether target binaries are the right place for those
things at all - IMHO that's something one wants to derive from the
source code  / .config's. At least in the cases I'm imagining, I don't
even have an actual kernel running on the corresponding target yet.
(eg. in crosscompile situations)

OTOH, a more pressing problem for me is identifying the right drivers
and corresponding config options (usually plural, as certain subsystems
have to be enabled, too) by hardware information like DT, ACPI, DMI,
PCI, etc. For now, I have to do that manually, which is pretty time
consuming.

In embedded world, we often have scenarios where we want a really
minimal kernel, but need to enable/disable certain hi-level peripherals
in the middle of the project (eg. "oh, we also need ethernet, but we
wanna drop usb"). There we'll have to find out what actual chip is,
its corresponding driver, required subsystems, etc, and also kick off
everything we don't need anymore.

I've thought about implementing some actual dependency tracking
(at least recording the auto-enabled symbols), but didn't expect that
to become practically usable anytime soon, so I went for a different
approach: writing a little tool that allows modeling hilevel features
and corresponding (potentially board-specific) config syms, so the whole
.config for certain board and usecase can be autogenerated by just some
small meta-configuration:

   https://github.com/metux/kmct

Maybe this could also help for your usecase ?


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 02:16:44PM +0200, Enrico Weigelt, metux IT consult wrote:
> On 28.06.19 20:40, Luis Chamberlain wrote:
> 
> Hi folks,
> 
> > The solution puts forward a mechanism to add a kconfig_symb where
> > we are 100% certain we have a direct module --> config mapping.
> Okay, but IIRC this will add more boilerplate those modules.

Just one module attribute.

> And I wonder whether target binaries are the right place for those
> things at all - IMHO that's something one wants to derive from the
> source code  / .config's.

For the use cases mentioned for why the module attribute is being
suggested it would help to not have to download kernel sources. The
only question we want to answer is: for the hardware components
present on this system, which configs options do I need to enable
to support these components?

> At least in the cases I'm imagining, I don't
> even have an actual kernel running on the corresponding target yet.
> (eg. in crosscompile situations)

Right, so this is a different use case. A *running* kernel provides
*more* information, specially if it had enabled on it modules which
enable hardware.

> OTOH, a more pressing problem for me is identifying the right drivers
> and corresponding config options (usually plural, as certain subsystems
> have to be enabled, too) by hardware information like DT, ACPI, DMI,
> PCI, etc.

Right, I'm familiar with the problem.

At least for virtualization we decided to support at least these two to
help:

  * make kvmconfig
  * make xenconfig

I'm not suggesting we need a respective target per major component you
have described. I'm mentioning these as examples of ways to also address
similar problems.

> For now, I have to do that manually, which is pretty time
> consuming.

Yes...

Similar problem would be found if one wanted to find a desirable kernel
config for a remote system. One should be able to somehow scrape some
hardware information, dump that to a file, and then somehow generate
a working config for that system.

The module attribute being suggested would enable at least one way
to gather some of the required config symbols: symbols for *hardware*
and where one can run a modern kernel, with many features / hardware
enabled already.

Addressing the problem of obtaining a kernel config for a system where
one cannot run any kernel should be possible, however this *is* a very
different problem.

However, folks producing embedded systems *do* / *should* have a lot of
knowledge of their systems, and so the type of scheme you have devised
seems sensible for it.

> In embedded world, we often have scenarios where we want a really
> minimal kernel, but need to enable/disable certain hi-level peripherals
> in the middle of the project (eg. "oh, we also need ethernet, but we
> wanna drop usb"). There we'll have to find out what actual chip is,
> its corresponding driver, required subsystems, etc, and also kick off
> everything we don't need anymore.

Right. One *should* be able to tell some tool, hey, here is the list of
my desirable .config options. Go and figure out what I need to make that
work and give me a resulting .config. Its not easy.

> I've thought about implementing some actual dependency tracking
> (at least recording the auto-enabled symbols), but didn't expect that
> to become practically usable anytime soon,

The ability to easily ask the kernel to enable the components needed
for a respective config option *is* very useful but indeed not easy.

This is not the only space where this problem exists. Similar problem
exists for distribution packages, and dependencies. Challenges have
been made for proper research towards these problems, and such research
has lead distributions to opt to enable some of these algorithms.

This begs the question if we could learn from similar efforts on Linux
for these sorts of questions. One possibility here is to evaluate the
prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
(MUSes) support, which should be be able to address thir problem. This
prospect is ongoing and currrent R&D is active, for details refer to:

https://kernelnewbies.org/KernelProjects/kconfig-sat

I'd recommend the mailing list, as that is where folks are following up
on, and there seems to be a call scheduled next Wednesday at 8:30am PST
about some new developments.

> so I went for a different
> approach: writing a little tool that allows modeling hilevel features
> and corresponding (potentially board-specific) config syms, so the whole
> .config for certain board and usecase can be autogenerated by just some
> small meta-configuration:
> 
>    https://github.com/metux/kmct
> 
> Maybe this could also help for your usecase ?

It certainly can be useful for components, ie, not hardware. But for
hardware a one-to-one mapping of one driver to one config would be of
much more use.

Based on a quick glance, I'd suggest that it would be great if could
have something similar upstream on linux. It would be unclear yet how to
structure this though. It would be wonderful if for instance kconfig
supported a way to group a major set of components under *one* config
symbol and then say: "I want this major component enabled" and then it'd
go and enable all the defaults which would be required for it. Its not
an easy problem, given though that there may be certain other symbols
which may conflict with your target component. An example is if you
wanted to enable PCI on a system which didn't support it. Because of
this, it seems you'd want *all* desirable configs and let a piece of
software figure out what you need / can enable. And.. this is precisely
where the SAT solver with MUSes could help...

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 02:16:44PM +0200, Enrico Weigelt, metux IT consult wrote:
> On 28.06.19 20:40, Luis Chamberlain wrote:
> 
> Hi folks,
> 
> > The solution puts forward a mechanism to add a kconfig_symb where
> > we are 100% certain we have a direct module --> config mapping.
> Okay, but IIRC this will add more boilerplate those modules.

Just one module attribute.

> And I wonder whether target binaries are the right place for those
> things at all - IMHO that's something one wants to derive from the
> source code  / .config's.

For the use cases mentioned for why the module attribute is being
suggested it would help to not have to download kernel sources. The
only question we want to answer is: for the hardware components
present on this system, which configs options do I need to enable
to support these components?

> At least in the cases I'm imagining, I don't
> even have an actual kernel running on the corresponding target yet.
> (eg. in crosscompile situations)

Right, so this is a different use case. A *running* kernel provides
*more* information, specially if it had enabled on it modules which
enable hardware.

> OTOH, a more pressing problem for me is identifying the right drivers
> and corresponding config options (usually plural, as certain subsystems
> have to be enabled, too) by hardware information like DT, ACPI, DMI,
> PCI, etc.

Right, I'm familiar with the problem.

At least for virtualization we decided to support at least these two to
help:

  * make kvmconfig
  * make xenconfig

I'm not suggesting we need a respective target per major component you
have described. I'm mentioning these as examples of ways to also address
similar problems.

> For now, I have to do that manually, which is pretty time
> consuming.

Yes...

Similar problem would be found if one wanted to find a desirable kernel
config for a remote system. One should be able to somehow scrape some
hardware information, dump that to a file, and then somehow generate
a working config for that system.

The module attribute being suggested would enable at least one way
to gather some of the required config symbols: symbols for *hardware*
and where one can run a modern kernel, with many features / hardware
enabled already.

Addressing the problem of obtaining a kernel config for a system where
one cannot run any kernel should be possible, however this *is* a very
different problem.

However, folks producing embedded systems *do* / *should* have a lot of
knowledge of their systems, and so the type of scheme you have devised
seems sensible for it.

> In embedded world, we often have scenarios where we want a really
> minimal kernel, but need to enable/disable certain hi-level peripherals
> in the middle of the project (eg. "oh, we also need ethernet, but we
> wanna drop usb"). There we'll have to find out what actual chip is,
> its corresponding driver, required subsystems, etc, and also kick off
> everything we don't need anymore.

Right. One *should* be able to tell some tool, hey, here is the list of
my desirable .config options. Go and figure out what I need to make that
work and give me a resulting .config. Its not easy.

> I've thought about implementing some actual dependency tracking
> (at least recording the auto-enabled symbols), but didn't expect that
> to become practically usable anytime soon,

The ability to easily ask the kernel to enable the components needed
for a respective config option *is* very useful but indeed not easy.

This is not the only space where this problem exists. Similar problem
exists for distribution packages, and dependencies. Challenges have
been made for proper research towards these problems, and such research
has lead distributions to opt to enable some of these algorithms.

This begs the question if we could learn from similar efforts on Linux
for these sorts of questions. One possibility here is to evaluate the
prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
(MUSes) support, which should be be able to address thir problem. This
prospect is ongoing and currrent R&D is active, for details refer to:

https://kernelnewbies.org/KernelProjects/kconfig-sat

I'd recommend the mailing list, as that is where folks are following up
on, and there seems to be a call scheduled next Wednesday at 8:30am PST
about some new developments.

> so I went for a different
> approach: writing a little tool that allows modeling hilevel features
> and corresponding (potentially board-specific) config syms, so the whole
> .config for certain board and usecase can be autogenerated by just some
> small meta-configuration:
> 
>    https://github.com/metux/kmct
> 
> Maybe this could also help for your usecase ?

It certainly can be useful for components, ie, not hardware. But for
hardware a one-to-one mapping of one driver to one config would be of
much more use.

Based on a quick glance, I'd suggest that it would be great if could
have something similar upstream on linux. It would be unclear yet how to
structure this though. It would be wonderful if for instance kconfig
supported a way to group a major set of components under *one* config
symbol and then say: "I want this major component enabled" and then it'd
go and enable all the defaults which would be required for it. Its not
an easy problem, given though that there may be certain other symbols
which may conflict with your target component. An example is if you
wanted to enable PCI on a system which didn't support it. Because of
this, it seems you'd want *all* desirable configs and let a piece of
software figure out what you need / can enable. And.. this is precisely
where the SAT solver with MUSes could help...

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Enrico Weigelt, metux IT consult]

On 03.07.19 19:35, Luis Chamberlain wrote:

Hi,

>> Okay, but IIRC this will add more boilerplate those modules.
> 
> Just one module attribute.

Yes, but still one per module. This raises the question whether
maintainers are willing to cope w/ tons of tiny patches for just
one line - for something that will take quite some time to become
actually useful (doesn't help much if only few drivers support it),
and is only helpful in a few use cases.

And to make it really useful, we also need some way to automatically
derive which other symbols to enable (subsystems, etc), w/o auto-
enabling stuff one doesn't need here. (are the defaults sane for
this usecase ?)

The main problem here, IMHO, is that the kconfig system doesn't really
know what makes up a module (it only knows that something w =y cant
depend on something thats =m).

So it smalls like we'd need some config language that really understands
things like modules, subsystems, arches, etc with their properties and
is used by both kconfig and kbuild. Then we could put all metadata there
instead of the current macro calls. At that point we could also put
things like match tables in here, which would solve the problem of
finding the right driver by hardware descriptions.

But that's really a *big* topic, it's not easy.

>> And I wonder whether target binaries are the right place for those
>> things at all - IMHO that's something one wants to derive from the
>> source code  / .config's.
> 
> For the use cases mentioned for why the module attribute is being
> suggested it would help to not have to download kernel sources. 

Are we still talking about compiling custom kernels ?
(how to do that w/o source code ?)

> The only question we want to answer is: for the hardware components
> present on this system, which configs options do I need to enable
> to support these components?

What else would one need that data, if not compiling a custom kernel
(which in turn needs the source) ?

> At least for virtualization we decided to support at least these two to
> help:
> 
>   * make kvmconfig
>   * make xenconfig

These two are rather simple. Most times there isn't much variance in
virtual hardware (unless one starts directly mapping in pci or usb
devices ...)

> Similar problem would be found if one wanted to find a desirable kernel
> config for a remote system. One should be able to somehow scrape some
> hardware information, dump that to a file, and then somehow generate
> a working config for that system.

Yes. That's actually pretty much the same usecase (in my case I'd also
have dts, lspci/lsusb output, etc)

> The module attribute being suggested would enable at least one way
> to gather some of the required config symbols: symbols for *hardware*
> and where one can run a modern kernel, with many features / hardware
> enabled already.

But only for a pretty specific usecase. I'm not opposed to this, but
I wonder whether maintainers are willing to accept that stuff for just
that specific usecase.

> However, folks producing embedded systems *do* / *should* have a lot of
> knowledge of their systems, and so the type of scheme you have devised
> seems sensible for it.

Usually we have (unless we need to do reverse engineering :o). But it's
a pretty time-consuming task. Especially if the requirements change
several times in the development or lifetime of a specific product.

For example "oh, we now need eth", "naah, we don't wanna use usb
anymore", "let's take a different SoM", ... not that have pretty
orthogonal sets of configs we need to maintain: hardware- and non-
hardware-related ones. And hardware-related ones can fall into different
categories like fixed-attached/onboard vs. hotpluggable ones.

Recently I had a case where the customer requested xattr support, so
I had to enable general xattr support as well as per-filesystem.
Pretty simple, but having lots of those cases quickly sums up. One of
the reasons why I've written my own little config generator.

>> In embedded world, we often have scenarios where we want a really
>> minimal kernel, but need to enable/disable certain hi-level peripherals
>> in the middle of the project (eg. "oh, we also need ethernet, but we
>> wanna drop usb"). There we'll have to find out what actual chip is,
>> its corresponding driver, required subsystems, etc, and also kick off
>> everything we don't need anymore.
> 
> Right. One *should* be able to tell some tool, hey, here is the list of
> my desirable .config options. Go and figure out what I need to make that
> work and give me a resulting .config. Its not easy.

I think I've already got into a pretty usable state - at least for my
projects. For now only supports a few boards and limited set of
features, but patches are always welcomed :)

>> I've thought about implementing some actual dependency tracking
>> (at least recording the auto-enabled symbols), but didn't expect that
>> to become practically usable anytime soon,
> 
> The ability to easily ask the kernel to enable the components needed
> for a respective config option *is* very useful but indeed not easy.

Yes, it would need to understand things like conditional definitions
to deduce that certain things need to be enabled first, before certain
drivers become choosable.

> This is not the only space where this problem exists. Similar problem
> exists for distribution packages, and dependencies. Challenges have
> been made for proper research towards these problems, and such research
> has lead distributions to opt to enable some of these algorithms.

The problem w/ dependencies is that there can be different types of
dependencies, as well as different types of software objects. Just
solving the expressions is only a part of the problem.

> This begs the question if we could learn from similar efforts on Linux
> for these sorts of questions. One possibility here is to evaluate the
> prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> (MUSes) support, which should be be able to address thir problem. This
> prospect is ongoing and currrent R&D is active, for details refer to:
> 
> https://kernelnewbies.org/KernelProjects/kconfig-sat

Good tip, I'll have a look at it.

> It certainly can be useful for components, ie, not hardware. But for
> hardware a one-to-one mapping of one driver to one config would be of
> much more use.

Unfortunately, we don't have this 1:1 mapping. Often drivers support
different sets of devices, depending on other factors, sometimes sub-
options (eg. different hw versions), sometimes depending on other
subsystems, sometimes arch-specific, etc, etc.

I think we should work towards that, but I doubt we'd reach that goal
anytime soon, and begs the question whether it's really worth all the
effort required for that.

> It would be wonderful if for instance kconfig
> supported a way to group a major set of components under *one* config
> symbol and then say: "I want this major component enabled" and then it'd
> go and enable all the defaults which would be required for it. 

Yes, thought about that, too. For example have syms for selecting whole
boards and features of them - a bit like this:

  --> Preconfigure for specific boards
      --> board A
      --> board B
      ...
  --> Enable board features
      --> Ethernet port
      --> Display
          --> Touch panel
      --> Audio
      ....

BUT: this would turn into maintenance hell, so I dropped that idea.

> An example is if you
> wanted to enable PCI on a system which didn't support it. Because of
> this, it seems you'd want *all* desirable configs and let a piece of
> software figure out what you need / can enable. And.. this is precisely
> where the SAT solver with MUSes could help...

Yes, but this piece of software first needs to know whether eg. PCI
is available on that HW. Oh, and things like PCI could be a dependency
as well as an feature on its own, depending on how you gonna use it.
(eg. if directly access from userland or VMs).


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Enrico Weigelt, metux IT consult]

On 03.07.19 19:35, Luis Chamberlain wrote:

Hi,

>> Okay, but IIRC this will add more boilerplate those modules.
> 
> Just one module attribute.

Yes, but still one per module. This raises the question whether
maintainers are willing to cope w/ tons of tiny patches for just
one line - for something that will take quite some time to become
actually useful (doesn't help much if only few drivers support it),
and is only helpful in a few use cases.

And to make it really useful, we also need some way to automatically
derive which other symbols to enable (subsystems, etc), w/o auto-
enabling stuff one doesn't need here. (are the defaults sane for
this usecase ?)

The main problem here, IMHO, is that the kconfig system doesn't really
know what makes up a module (it only knows that something w =y cant
depend on something thats =m).

So it smalls like we'd need some config language that really understands
things like modules, subsystems, arches, etc with their properties and
is used by both kconfig and kbuild. Then we could put all metadata there
instead of the current macro calls. At that point we could also put
things like match tables in here, which would solve the problem of
finding the right driver by hardware descriptions.

But that's really a *big* topic, it's not easy.

>> And I wonder whether target binaries are the right place for those
>> things at all - IMHO that's something one wants to derive from the
>> source code  / .config's.
> 
> For the use cases mentioned for why the module attribute is being
> suggested it would help to not have to download kernel sources. 

Are we still talking about compiling custom kernels ?
(how to do that w/o source code ?)

> The only question we want to answer is: for the hardware components
> present on this system, which configs options do I need to enable
> to support these components?

What else would one need that data, if not compiling a custom kernel
(which in turn needs the source) ?

> At least for virtualization we decided to support at least these two to
> help:
> 
>   * make kvmconfig
>   * make xenconfig

These two are rather simple. Most times there isn't much variance in
virtual hardware (unless one starts directly mapping in pci or usb
devices ...)

> Similar problem would be found if one wanted to find a desirable kernel
> config for a remote system. One should be able to somehow scrape some
> hardware information, dump that to a file, and then somehow generate
> a working config for that system.

Yes. That's actually pretty much the same usecase (in my case I'd also
have dts, lspci/lsusb output, etc)

> The module attribute being suggested would enable at least one way
> to gather some of the required config symbols: symbols for *hardware*
> and where one can run a modern kernel, with many features / hardware
> enabled already.

But only for a pretty specific usecase. I'm not opposed to this, but
I wonder whether maintainers are willing to accept that stuff for just
that specific usecase.

> However, folks producing embedded systems *do* / *should* have a lot of
> knowledge of their systems, and so the type of scheme you have devised
> seems sensible for it.

Usually we have (unless we need to do reverse engineering :o). But it's
a pretty time-consuming task. Especially if the requirements change
several times in the development or lifetime of a specific product.

For example "oh, we now need eth", "naah, we don't wanna use usb
anymore", "let's take a different SoM", ... not that have pretty
orthogonal sets of configs we need to maintain: hardware- and non-
hardware-related ones. And hardware-related ones can fall into different
categories like fixed-attached/onboard vs. hotpluggable ones.

Recently I had a case where the customer requested xattr support, so
I had to enable general xattr support as well as per-filesystem.
Pretty simple, but having lots of those cases quickly sums up. One of
the reasons why I've written my own little config generator.

>> In embedded world, we often have scenarios where we want a really
>> minimal kernel, but need to enable/disable certain hi-level peripherals
>> in the middle of the project (eg. "oh, we also need ethernet, but we
>> wanna drop usb"). There we'll have to find out what actual chip is,
>> its corresponding driver, required subsystems, etc, and also kick off
>> everything we don't need anymore.
> 
> Right. One *should* be able to tell some tool, hey, here is the list of
> my desirable .config options. Go and figure out what I need to make that
> work and give me a resulting .config. Its not easy.

I think I've already got into a pretty usable state - at least for my
projects. For now only supports a few boards and limited set of
features, but patches are always welcomed :)

>> I've thought about implementing some actual dependency tracking
>> (at least recording the auto-enabled symbols), but didn't expect that
>> to become practically usable anytime soon,
> 
> The ability to easily ask the kernel to enable the components needed
> for a respective config option *is* very useful but indeed not easy.

Yes, it would need to understand things like conditional definitions
to deduce that certain things need to be enabled first, before certain
drivers become choosable.

> This is not the only space where this problem exists. Similar problem
> exists for distribution packages, and dependencies. Challenges have
> been made for proper research towards these problems, and such research
> has lead distributions to opt to enable some of these algorithms.

The problem w/ dependencies is that there can be different types of
dependencies, as well as different types of software objects. Just
solving the expressions is only a part of the problem.

> This begs the question if we could learn from similar efforts on Linux
> for these sorts of questions. One possibility here is to evaluate the
> prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> (MUSes) support, which should be be able to address thir problem. This
> prospect is ongoing and currrent R&D is active, for details refer to:
> 
> https://kernelnewbies.org/KernelProjects/kconfig-sat

Good tip, I'll have a look at it.

> It certainly can be useful for components, ie, not hardware. But for
> hardware a one-to-one mapping of one driver to one config would be of
> much more use.

Unfortunately, we don't have this 1:1 mapping. Often drivers support
different sets of devices, depending on other factors, sometimes sub-
options (eg. different hw versions), sometimes depending on other
subsystems, sometimes arch-specific, etc, etc.

I think we should work towards that, but I doubt we'd reach that goal
anytime soon, and begs the question whether it's really worth all the
effort required for that.

> It would be wonderful if for instance kconfig
> supported a way to group a major set of components under *one* config
> symbol and then say: "I want this major component enabled" and then it'd
> go and enable all the defaults which would be required for it. 

Yes, thought about that, too. For example have syms for selecting whole
boards and features of them - a bit like this:

  --> Preconfigure for specific boards
      --> board A
      --> board B
      ...
  --> Enable board features
      --> Ethernet port
      --> Display
          --> Touch panel
      --> Audio
      ....

BUT: this would turn into maintenance hell, so I dropped that idea.

> An example is if you
> wanted to enable PCI on a system which didn't support it. Because of
> this, it seems you'd want *all* desirable configs and let a piece of
> software figure out what you need / can enable. And.. this is precisely
> where the SAT solver with MUSes could help...

Yes, but this piece of software first needs to know whether eg. PCI
is available on that HW. Oh, and things like PCI could be a dependency
as well as an feature on its own, depending on how you gonna use it.
(eg. if directly access from userland or VMs).


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 09:31:33PM +0200, Enrico Weigelt, metux IT consult wrote:
> On 03.07.19 19:35, Luis Chamberlain wrote:
> 
> Hi,
> 
> >> Okay, but IIRC this will add more boilerplate those modules.
> > 
> > Just one module attribute.
> 
> Yes, but still one per module. This raises the question whether
> maintainers are willing to cope w/ tons of tiny patches for just
> one line - for something that will take quite some time to become
> actually useful (doesn't help much if only few drivers support it),
> and is only helpful in a few use cases.

This can be wrapped in kconfig and disabled by default.

> And to make it really useful, we also need some way to automatically
> derive which other symbols to enable (subsystems, etc), w/o auto-
> enabling stuff one doesn't need here. (are the defaults sane for
> this usecase ?)

No, that's a separate problem.
> 
> The main problem here, IMHO, is that the kconfig system doesn't really
> know what makes up a module (it only knows that something w =y cant
> depend on something thats =m).
> 
> So it smalls like we'd need some config language that really understands
> things like modules, subsystems, arches, etc with their properties and
> is used by both kconfig and kbuild. Then we could put all metadata there
> instead of the current macro calls. At that point we could also put
> things like match tables in here, which would solve the problem of
> finding the right driver by hardware descriptions.
> 
> But that's really a *big* topic, it's not easy.

Christoph's sugggestion goes along the lines with addressing some of
this, yet 2 years havegone by and no one is working on it and its hugely
intrusive. I'd welcome the patches.

> >> And I wonder whether target binaries are the right place for those
> >> things at all - IMHO that's something one wants to derive from the
> >> source code  / .config's.
> > 
> > For the use cases mentioned for why the module attribute is being
> > suggested it would help to not have to download kernel sources. 
> 
> Are we still talking about compiling custom kernels ?
> (how to do that w/o source code ?)

There is a difference between getting kernel sources for your current
running kernel Vs getting kernel sources from say Linus' tree or the
stable tree, or whatever subsystem you work on. I'm saying that with
a module attribute being present you would *not* have to get the kernel
sources for the current kernel you are running.

> > The only question we want to answer is: for the hardware components
> > present on this system, which configs options do I need to enable
> > to support these components?
> 
> What else would one need that data, if not compiling a custom kernel
> (which in turn needs the source) ?

One is compiling a custom kernel.

> > At least for virtualization we decided to support at least these two to
> > help:
> > 
> >   * make kvmconfig
> >   * make xenconfig
> 
> These two are rather simple. Most times there isn't much variance in
> virtual hardware (unless one starts directly mapping in pci or usb
> devices ...)

Actually no, there is huge variance on the qemu level and that problem
is not solved by the above.

> > Similar problem would be found if one wanted to find a desirable kernel
> > config for a remote system. One should be able to somehow scrape some
> > hardware information, dump that to a file, and then somehow generate
> > a working config for that system.
> 
> Yes. That's actually pretty much the same usecase (in my case I'd also
> have dts, lspci/lsusb output, etc)
> 
> > The module attribute being suggested would enable at least one way
> > to gather some of the required config symbols: symbols for *hardware*
> > and where one can run a modern kernel, with many features / hardware
> > enabled already.
> 
> But only for a pretty specific usecase.

Two: build time, and backports.

> I'm not opposed to this, but
> I wonder whether maintainers are willing to accept that stuff for just
> that specific usecase.

This is why we are discussing this on this thread.

Since we have 'make localmodconfig' already upstream, and since this
would help both users of that and backports I'd argue it makes sense
upstream. Otherwise I find it that having upstream 'make localmodconfig'
but not wanting to improve that problem space rather odd.

> > However, folks producing embedded systems *do* / *should* have a lot of
> > knowledge of their systems, and so the type of scheme you have devised
> > seems sensible for it.
> 
> Usually we have (unless we need to do reverse engineering :o). But it's
> a pretty time-consuming task. Especially if the requirements change
> several times in the development or lifetime of a specific product.
> 
> For example "oh, we now need eth", "naah, we don't wanna use usb
> anymore", "let's take a different SoM", ... not that have pretty
> orthogonal sets of configs we need to maintain: hardware- and non-
> hardware-related ones. And hardware-related ones can fall into different
> categories like fixed-attached/onboard vs. hotpluggable ones.
> 
> Recently I had a case where the customer requested xattr support, so
> I had to enable general xattr support as well as per-filesystem.
> Pretty simple, but having lots of those cases quickly sums up. One of
> the reasons why I've written my own little config generator.

I agree that the problem space you are dealing with should be made
easier.

> >> In embedded world, we often have scenarios where we want a really
> >> minimal kernel, but need to enable/disable certain hi-level peripherals
> >> in the middle of the project (eg. "oh, we also need ethernet, but we
> >> wanna drop usb"). There we'll have to find out what actual chip is,
> >> its corresponding driver, required subsystems, etc, and also kick off
> >> everything we don't need anymore.
> > 
> > Right. One *should* be able to tell some tool, hey, here is the list of
> > my desirable .config options. Go and figure out what I need to make that
> > work and give me a resulting .config. Its not easy.
> 
> I think I've already got into a pretty usable state - at least for my
> projects. For now only supports a few boards and limited set of
> features, but patches are always welcomed :)

A solution upstream would be better ;)

> >> I've thought about implementing some actual dependency tracking
> >> (at least recording the auto-enabled symbols), but didn't expect that
> >> to become practically usable anytime soon,
> > 
> > The ability to easily ask the kernel to enable the components needed
> > for a respective config option *is* very useful but indeed not easy.
> 
> Yes, it would need to understand things like conditional definitions
> to deduce that certain things need to be enabled first, before certain
> drivers become choosable.
> 
> > This is not the only space where this problem exists. Similar problem
> > exists for distribution packages, and dependencies. Challenges have
> > been made for proper research towards these problems, and such research
> > has lead distributions to opt to enable some of these algorithms.
> 
> The problem w/ dependencies is that there can be different types of
> dependencies, as well as different types of software objects. Just
> solving the expressions is only a part of the problem.
> 
> > This begs the question if we could learn from similar efforts on Linux
> > for these sorts of questions. One possibility here is to evaluate the
> > prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> > (MUSes) support, which should be be able to address thir problem. This
> > prospect is ongoing and currrent R&D is active, for details refer to:
> > 
> > https://kernelnewbies.org/KernelProjects/kconfig-sat
> 
> Good tip, I'll have a look at it.
> 
> > It certainly can be useful for components, ie, not hardware. But for
> > hardware a one-to-one mapping of one driver to one config would be of
> > much more use.
> 
> Unfortunately, we don't have this 1:1 mapping. Often drivers support
> different sets of devices, depending on other factors, sometimes sub-
> options (eg. different hw versions), sometimes depending on other
> subsystems, sometimes arch-specific, etc, etc.
> 
> I think we should work towards that, but I doubt we'd reach that goal
> anytime soon, and begs the question whether it's really worth all the
> effort required for that.

Code speaks, and fortunately its being worked on.

> > It would be wonderful if for instance kconfig
> > supported a way to group a major set of components under *one* config
> > symbol and then say: "I want this major component enabled" and then it'd
> > go and enable all the defaults which would be required for it. 
> 
> Yes, thought about that, too. For example have syms for selecting whole
> boards and features of them - a bit like this:
> 
>   --> Preconfigure for specific boards
>       --> board A
>       --> board B
>       ...
>   --> Enable board features
>       --> Ethernet port
>       --> Display
>           --> Touch panel
>       --> Audio
>       ....
> 
> BUT: this would turn into maintenance hell, so I dropped that idea.
> 
> > An example is if you
> > wanted to enable PCI on a system which didn't support it. Because of
> > this, it seems you'd want *all* desirable configs and let a piece of
> > software figure out what you need / can enable. And.. this is precisely
> > where the SAT solver with MUSes could help...
> 
> Yes, but this piece of software first needs to know whether eg. PCI
> is available on that HW. Oh, and things like PCI could be a dependency
> as well as an feature on its own, depending on how you gonna use it.
> (eg. if directly access from userland or VMs).

Yes, but that is a separate problem. The way 'make localmodconfig'
addresses this is by keeping as modules things which are already
modules. And keeping things it found as =y as buit-in in your kernel as
well.

  Luis
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Luis Chamberlain]

On Wed, Jul 03, 2019 at 09:31:33PM +0200, Enrico Weigelt, metux IT consult wrote:
> On 03.07.19 19:35, Luis Chamberlain wrote:
> 
> Hi,
> 
> >> Okay, but IIRC this will add more boilerplate those modules.
> > 
> > Just one module attribute.
> 
> Yes, but still one per module. This raises the question whether
> maintainers are willing to cope w/ tons of tiny patches for just
> one line - for something that will take quite some time to become
> actually useful (doesn't help much if only few drivers support it),
> and is only helpful in a few use cases.

This can be wrapped in kconfig and disabled by default.

> And to make it really useful, we also need some way to automatically
> derive which other symbols to enable (subsystems, etc), w/o auto-
> enabling stuff one doesn't need here. (are the defaults sane for
> this usecase ?)

No, that's a separate problem.
> 
> The main problem here, IMHO, is that the kconfig system doesn't really
> know what makes up a module (it only knows that something w =y cant
> depend on something thats =m).
> 
> So it smalls like we'd need some config language that really understands
> things like modules, subsystems, arches, etc with their properties and
> is used by both kconfig and kbuild. Then we could put all metadata there
> instead of the current macro calls. At that point we could also put
> things like match tables in here, which would solve the problem of
> finding the right driver by hardware descriptions.
> 
> But that's really a *big* topic, it's not easy.

Christoph's sugggestion goes along the lines with addressing some of
this, yet 2 years havegone by and no one is working on it and its hugely
intrusive. I'd welcome the patches.

> >> And I wonder whether target binaries are the right place for those
> >> things at all - IMHO that's something one wants to derive from the
> >> source code  / .config's.
> > 
> > For the use cases mentioned for why the module attribute is being
> > suggested it would help to not have to download kernel sources. 
> 
> Are we still talking about compiling custom kernels ?
> (how to do that w/o source code ?)

There is a difference between getting kernel sources for your current
running kernel Vs getting kernel sources from say Linus' tree or the
stable tree, or whatever subsystem you work on. I'm saying that with
a module attribute being present you would *not* have to get the kernel
sources for the current kernel you are running.

> > The only question we want to answer is: for the hardware components
> > present on this system, which configs options do I need to enable
> > to support these components?
> 
> What else would one need that data, if not compiling a custom kernel
> (which in turn needs the source) ?

One is compiling a custom kernel.

> > At least for virtualization we decided to support at least these two to
> > help:
> > 
> >   * make kvmconfig
> >   * make xenconfig
> 
> These two are rather simple. Most times there isn't much variance in
> virtual hardware (unless one starts directly mapping in pci or usb
> devices ...)

Actually no, there is huge variance on the qemu level and that problem
is not solved by the above.

> > Similar problem would be found if one wanted to find a desirable kernel
> > config for a remote system. One should be able to somehow scrape some
> > hardware information, dump that to a file, and then somehow generate
> > a working config for that system.
> 
> Yes. That's actually pretty much the same usecase (in my case I'd also
> have dts, lspci/lsusb output, etc)
> 
> > The module attribute being suggested would enable at least one way
> > to gather some of the required config symbols: symbols for *hardware*
> > and where one can run a modern kernel, with many features / hardware
> > enabled already.
> 
> But only for a pretty specific usecase.

Two: build time, and backports.

> I'm not opposed to this, but
> I wonder whether maintainers are willing to accept that stuff for just
> that specific usecase.

This is why we are discussing this on this thread.

Since we have 'make localmodconfig' already upstream, and since this
would help both users of that and backports I'd argue it makes sense
upstream. Otherwise I find it that having upstream 'make localmodconfig'
but not wanting to improve that problem space rather odd.

> > However, folks producing embedded systems *do* / *should* have a lot of
> > knowledge of their systems, and so the type of scheme you have devised
> > seems sensible for it.
> 
> Usually we have (unless we need to do reverse engineering :o). But it's
> a pretty time-consuming task. Especially if the requirements change
> several times in the development or lifetime of a specific product.
> 
> For example "oh, we now need eth", "naah, we don't wanna use usb
> anymore", "let's take a different SoM", ... not that have pretty
> orthogonal sets of configs we need to maintain: hardware- and non-
> hardware-related ones. And hardware-related ones can fall into different
> categories like fixed-attached/onboard vs. hotpluggable ones.
> 
> Recently I had a case where the customer requested xattr support, so
> I had to enable general xattr support as well as per-filesystem.
> Pretty simple, but having lots of those cases quickly sums up. One of
> the reasons why I've written my own little config generator.

I agree that the problem space you are dealing with should be made
easier.

> >> In embedded world, we often have scenarios where we want a really
> >> minimal kernel, but need to enable/disable certain hi-level peripherals
> >> in the middle of the project (eg. "oh, we also need ethernet, but we
> >> wanna drop usb"). There we'll have to find out what actual chip is,
> >> its corresponding driver, required subsystems, etc, and also kick off
> >> everything we don't need anymore.
> > 
> > Right. One *should* be able to tell some tool, hey, here is the list of
> > my desirable .config options. Go and figure out what I need to make that
> > work and give me a resulting .config. Its not easy.
> 
> I think I've already got into a pretty usable state - at least for my
> projects. For now only supports a few boards and limited set of
> features, but patches are always welcomed :)

A solution upstream would be better ;)

> >> I've thought about implementing some actual dependency tracking
> >> (at least recording the auto-enabled symbols), but didn't expect that
> >> to become practically usable anytime soon,
> > 
> > The ability to easily ask the kernel to enable the components needed
> > for a respective config option *is* very useful but indeed not easy.
> 
> Yes, it would need to understand things like conditional definitions
> to deduce that certain things need to be enabled first, before certain
> drivers become choosable.
> 
> > This is not the only space where this problem exists. Similar problem
> > exists for distribution packages, and dependencies. Challenges have
> > been made for proper research towards these problems, and such research
> > has lead distributions to opt to enable some of these algorithms.
> 
> The problem w/ dependencies is that there can be different types of
> dependencies, as well as different types of software objects. Just
> solving the expressions is only a part of the problem.
> 
> > This begs the question if we could learn from similar efforts on Linux
> > for these sorts of questions. One possibility here is to evaluate the
> > prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> > (MUSes) support, which should be be able to address thir problem. This
> > prospect is ongoing and currrent R&D is active, for details refer to:
> > 
> > https://kernelnewbies.org/KernelProjects/kconfig-sat
> 
> Good tip, I'll have a look at it.
> 
> > It certainly can be useful for components, ie, not hardware. But for
> > hardware a one-to-one mapping of one driver to one config would be of
> > much more use.
> 
> Unfortunately, we don't have this 1:1 mapping. Often drivers support
> different sets of devices, depending on other factors, sometimes sub-
> options (eg. different hw versions), sometimes depending on other
> subsystems, sometimes arch-specific, etc, etc.
> 
> I think we should work towards that, but I doubt we'd reach that goal
> anytime soon, and begs the question whether it's really worth all the
> effort required for that.

Code speaks, and fortunately its being worked on.

> > It would be wonderful if for instance kconfig
> > supported a way to group a major set of components under *one* config
> > symbol and then say: "I want this major component enabled" and then it'd
> > go and enable all the defaults which would be required for it. 
> 
> Yes, thought about that, too. For example have syms for selecting whole
> boards and features of them - a bit like this:
> 
>   --> Preconfigure for specific boards
>       --> board A
>       --> board B
>       ...
>   --> Enable board features
>       --> Ethernet port
>       --> Display
>           --> Touch panel
>       --> Audio
>       ....
> 
> BUT: this would turn into maintenance hell, so I dropped that idea.
> 
> > An example is if you
> > wanted to enable PCI on a system which didn't support it. Because of
> > this, it seems you'd want *all* desirable configs and let a piece of
> > software figure out what you need / can enable. And.. this is precisely
> > where the SAT solver with MUSes could help...
> 
> Yes, but this piece of software first needs to know whether eg. PCI
> is available on that HW. Oh, and things like PCI could be a dependency
> as well as an feature on its own, depending on how you gonna use it.
> (eg. if directly access from userland or VMs).

Yes, but that is a separate problem. The way 'make localmodconfig'
addresses this is by keeping as modules things which are already
modules. And keeping things it found as =y as buit-in in your kernel as
well.

  Luis

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Brendan Higgins]

On Wed, Jul 3, 2019 at 3:42 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Wed, Jul 03, 2019 at 09:31:33PM +0200, Enrico Weigelt, metux IT consult wrote:
> > On 03.07.19 19:35, Luis Chamberlain wrote:
> >
> > Hi,
> >
> > >> Okay, but IIRC this will add more boilerplate those modules.
> > >
> > > Just one module attribute.
> >
> > Yes, but still one per module. This raises the question whether
> > maintainers are willing to cope w/ tons of tiny patches for just
> > one line - for something that will take quite some time to become
> > actually useful (doesn't help much if only few drivers support it),
> > and is only helpful in a few use cases.
>
> This can be wrapped in kconfig and disabled by default.
>
> > And to make it really useful, we also need some way to automatically
> > derive which other symbols to enable (subsystems, etc), w/o auto-
> > enabling stuff one doesn't need here. (are the defaults sane for
> > this usecase ?)
>
> No, that's a separate problem.

I am interested in this use case, but I agree that this is a separate issue.

Luis, I saw you cc'ed me on an email with the SAT solver people and
that they are still working on this. Enrico, want me to CC you on that
and we can continue this discussion there?

> > The main problem here, IMHO, is that the kconfig system doesn't really
> > know what makes up a module (it only knows that something w =y cant
> > depend on something thats =m).
> >
> > So it smalls like we'd need some config language that really understands
> > things like modules, subsystems, arches, etc with their properties and
> > is used by both kconfig and kbuild. Then we could put all metadata there
> > instead of the current macro calls. At that point we could also put
> > things like match tables in here, which would solve the problem of
> > finding the right driver by hardware descriptions.
> >
> > But that's really a *big* topic, it's not easy.
>
> Christoph's sugggestion goes along the lines with addressing some of
> this, yet 2 years havegone by and no one is working on it and its hugely
> intrusive. I'd welcome the patches.

I would like to see those. I would also very much like to see code
dependencies, not just kconfig dependencies, tracked in Kbuild
somehow.

> > >> And I wonder whether target binaries are the right place for those
> > >> things at all - IMHO that's something one wants to derive from the
> > >> source code  / .config's.
> > >
> > > For the use cases mentioned for why the module attribute is being
> > > suggested it would help to not have to download kernel sources.
> >
> > Are we still talking about compiling custom kernels ?
> > (how to do that w/o source code ?)
>
> There is a difference between getting kernel sources for your current
> running kernel Vs getting kernel sources from say Linus' tree or the
> stable tree, or whatever subsystem you work on. I'm saying that with
> a module attribute being present you would *not* have to get the kernel
> sources for the current kernel you are running.
>
> > > The only question we want to answer is: for the hardware components
> > > present on this system, which configs options do I need to enable
> > > to support these components?
> >
> > What else would one need that data, if not compiling a custom kernel
> > (which in turn needs the source) ?
>
> One is compiling a custom kernel.
>
> > > At least for virtualization we decided to support at least these two to
> > > help:
> > >
> > >   * make kvmconfig
> > >   * make xenconfig
> >
> > These two are rather simple. Most times there isn't much variance in
> > virtual hardware (unless one starts directly mapping in pci or usb
> > devices ...)
>
> Actually no, there is huge variance on the qemu level and that problem
> is not solved by the above.
>
> > > Similar problem would be found if one wanted to find a desirable kernel
> > > config for a remote system. One should be able to somehow scrape some
> > > hardware information, dump that to a file, and then somehow generate
> > > a working config for that system.
> >
> > Yes. That's actually pretty much the same usecase (in my case I'd also
> > have dts, lspci/lsusb output, etc)
> >
> > > The module attribute being suggested would enable at least one way
> > > to gather some of the required config symbols: symbols for *hardware*
> > > and where one can run a modern kernel, with many features / hardware
> > > enabled already.
> >
> > But only for a pretty specific usecase.
>
> Two: build time, and backports.

I think testing is another potential use case. Admittedly, I might
want something more general, but the general idea of being able to map
a config to code is super useful for automated testing.

For example, I want to run all tests. I enable all configs,cool. But
how do I know that the desired code under test actually ran?

Another example, okay I ran my tests and some of them failed. All my
test debug information is all mixed together and there is a lot of
noise present, also I am afraid that something else not part of the
test influenced the test or code under test behavior. So I want to
rerun the failed tests and the code that they test and nothing else.

> > I'm not opposed to this, but
> > I wonder whether maintainers are willing to accept that stuff for just
> > that specific usecase.
>
> This is why we are discussing this on this thread.
>
> Since we have 'make localmodconfig' already upstream, and since this
> would help both users of that and backports I'd argue it makes sense
> upstream. Otherwise I find it that having upstream 'make localmodconfig'
> but not wanting to improve that problem space rather odd.
>
> > > However, folks producing embedded systems *do* / *should* have a lot of
> > > knowledge of their systems, and so the type of scheme you have devised
> > > seems sensible for it.
> >
> > Usually we have (unless we need to do reverse engineering :o). But it's
> > a pretty time-consuming task. Especially if the requirements change
> > several times in the development or lifetime of a specific product.
> >
> > For example "oh, we now need eth", "naah, we don't wanna use usb
> > anymore", "let's take a different SoM", ... not that have pretty
> > orthogonal sets of configs we need to maintain: hardware- and non-
> > hardware-related ones. And hardware-related ones can fall into different
> > categories like fixed-attached/onboard vs. hotpluggable ones.
> >
> > Recently I had a case where the customer requested xattr support, so
> > I had to enable general xattr support as well as per-filesystem.
> > Pretty simple, but having lots of those cases quickly sums up. One of
> > the reasons why I've written my own little config generator.
>
> I agree that the problem space you are dealing with should be made
> easier.
>
> > >> In embedded world, we often have scenarios where we want a really
> > >> minimal kernel, but need to enable/disable certain hi-level peripherals
> > >> in the middle of the project (eg. "oh, we also need ethernet, but we
> > >> wanna drop usb"). There we'll have to find out what actual chip is,
> > >> its corresponding driver, required subsystems, etc, and also kick off
> > >> everything we don't need anymore.
> > >
> > > Right. One *should* be able to tell some tool, hey, here is the list of
> > > my desirable .config options. Go and figure out what I need to make that
> > > work and give me a resulting .config. Its not easy.
> >
> > I think I've already got into a pretty usable state - at least for my
> > projects. For now only supports a few boards and limited set of
> > features, but patches are always welcomed :)
>
> A solution upstream would be better ;)
>
> > >> I've thought about implementing some actual dependency tracking
> > >> (at least recording the auto-enabled symbols), but didn't expect that
> > >> to become practically usable anytime soon,
> > >
> > > The ability to easily ask the kernel to enable the components needed
> > > for a respective config option *is* very useful but indeed not easy.
> >
> > Yes, it would need to understand things like conditional definitions
> > to deduce that certain things need to be enabled first, before certain
> > drivers become choosable.
> >
> > > This is not the only space where this problem exists. Similar problem
> > > exists for distribution packages, and dependencies. Challenges have
> > > been made for proper research towards these problems, and such research
> > > has lead distributions to opt to enable some of these algorithms.
> >
> > The problem w/ dependencies is that there can be different types of
> > dependencies, as well as different types of software objects. Just
> > solving the expressions is only a part of the problem.
> >
> > > This begs the question if we could learn from similar efforts on Linux
> > > for these sorts of questions. One possibility here is to evaluate the
> > > prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> > > (MUSes) support, which should be be able to address thir problem. This
> > > prospect is ongoing and currrent R&D is active, for details refer to:
> > >
> > > https://kernelnewbies.org/KernelProjects/kconfig-sat
> >
> > Good tip, I'll have a look at it.
> >
> > > It certainly can be useful for components, ie, not hardware. But for
> > > hardware a one-to-one mapping of one driver to one config would be of
> > > much more use.
> >
> > Unfortunately, we don't have this 1:1 mapping. Often drivers support
> > different sets of devices, depending on other factors, sometimes sub-
> > options (eg. different hw versions), sometimes depending on other
> > subsystems, sometimes arch-specific, etc, etc.
> >
> > I think we should work towards that, but I doubt we'd reach that goal
> > anytime soon, and begs the question whether it's really worth all the
> > effort required for that.
>
> Code speaks, and fortunately its being worked on.
>
> > > It would be wonderful if for instance kconfig
> > > supported a way to group a major set of components under *one* config
> > > symbol and then say: "I want this major component enabled" and then it'd
> > > go and enable all the defaults which would be required for it.
> >
> > Yes, thought about that, too. For example have syms for selecting whole
> > boards and features of them - a bit like this:
> >
> >   --> Preconfigure for specific boards
> >       --> board A
> >       --> board B
> >       ...
> >   --> Enable board features
> >       --> Ethernet port
> >       --> Display
> >           --> Touch panel
> >       --> Audio
> >       ....
> >
> > BUT: this would turn into maintenance hell, so I dropped that idea.

I wonder if that would work for the testing scenario? I don't think it
is unreasonable for a test owner to provide a defconfig that makes it
possible to run their test. We could then merge these together to
create a kconfig to run all desired tests. Doesn't address all the
issues I mentioned here, but it's a start.

> > > An example is if you
> > > wanted to enable PCI on a system which didn't support it. Because of
> > > this, it seems you'd want *all* desirable configs and let a piece of
> > > software figure out what you need / can enable. And.. this is precisely
> > > where the SAT solver with MUSes could help...
> >
> > Yes, but this piece of software first needs to know whether eg. PCI
> > is available on that HW. Oh, and things like PCI could be a dependency
> > as well as an feature on its own, depending on how you gonna use it.
> > (eg. if directly access from userland or VMs).
>
> Yes, but that is a separate problem. The way 'make localmodconfig'
> addresses this is by keeping as modules things which are already
> modules. And keeping things it found as =y as buit-in in your kernel as
> well.

Cheers!
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Brendan Higgins]

On Wed, Jul 3, 2019 at 3:42 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Wed, Jul 03, 2019 at 09:31:33PM +0200, Enrico Weigelt, metux IT consult wrote:
> > On 03.07.19 19:35, Luis Chamberlain wrote:
> >
> > Hi,
> >
> > >> Okay, but IIRC this will add more boilerplate those modules.
> > >
> > > Just one module attribute.
> >
> > Yes, but still one per module. This raises the question whether
> > maintainers are willing to cope w/ tons of tiny patches for just
> > one line - for something that will take quite some time to become
> > actually useful (doesn't help much if only few drivers support it),
> > and is only helpful in a few use cases.
>
> This can be wrapped in kconfig and disabled by default.
>
> > And to make it really useful, we also need some way to automatically
> > derive which other symbols to enable (subsystems, etc), w/o auto-
> > enabling stuff one doesn't need here. (are the defaults sane for
> > this usecase ?)
>
> No, that's a separate problem.

I am interested in this use case, but I agree that this is a separate issue.

Luis, I saw you cc'ed me on an email with the SAT solver people and
that they are still working on this. Enrico, want me to CC you on that
and we can continue this discussion there?

> > The main problem here, IMHO, is that the kconfig system doesn't really
> > know what makes up a module (it only knows that something w =y cant
> > depend on something thats =m).
> >
> > So it smalls like we'd need some config language that really understands
> > things like modules, subsystems, arches, etc with their properties and
> > is used by both kconfig and kbuild. Then we could put all metadata there
> > instead of the current macro calls. At that point we could also put
> > things like match tables in here, which would solve the problem of
> > finding the right driver by hardware descriptions.
> >
> > But that's really a *big* topic, it's not easy.
>
> Christoph's sugggestion goes along the lines with addressing some of
> this, yet 2 years havegone by and no one is working on it and its hugely
> intrusive. I'd welcome the patches.

I would like to see those. I would also very much like to see code
dependencies, not just kconfig dependencies, tracked in Kbuild
somehow.

> > >> And I wonder whether target binaries are the right place for those
> > >> things at all - IMHO that's something one wants to derive from the
> > >> source code  / .config's.
> > >
> > > For the use cases mentioned for why the module attribute is being
> > > suggested it would help to not have to download kernel sources.
> >
> > Are we still talking about compiling custom kernels ?
> > (how to do that w/o source code ?)
>
> There is a difference between getting kernel sources for your current
> running kernel Vs getting kernel sources from say Linus' tree or the
> stable tree, or whatever subsystem you work on. I'm saying that with
> a module attribute being present you would *not* have to get the kernel
> sources for the current kernel you are running.
>
> > > The only question we want to answer is: for the hardware components
> > > present on this system, which configs options do I need to enable
> > > to support these components?
> >
> > What else would one need that data, if not compiling a custom kernel
> > (which in turn needs the source) ?
>
> One is compiling a custom kernel.
>
> > > At least for virtualization we decided to support at least these two to
> > > help:
> > >
> > >   * make kvmconfig
> > >   * make xenconfig
> >
> > These two are rather simple. Most times there isn't much variance in
> > virtual hardware (unless one starts directly mapping in pci or usb
> > devices ...)
>
> Actually no, there is huge variance on the qemu level and that problem
> is not solved by the above.
>
> > > Similar problem would be found if one wanted to find a desirable kernel
> > > config for a remote system. One should be able to somehow scrape some
> > > hardware information, dump that to a file, and then somehow generate
> > > a working config for that system.
> >
> > Yes. That's actually pretty much the same usecase (in my case I'd also
> > have dts, lspci/lsusb output, etc)
> >
> > > The module attribute being suggested would enable at least one way
> > > to gather some of the required config symbols: symbols for *hardware*
> > > and where one can run a modern kernel, with many features / hardware
> > > enabled already.
> >
> > But only for a pretty specific usecase.
>
> Two: build time, and backports.

I think testing is another potential use case. Admittedly, I might
want something more general, but the general idea of being able to map
a config to code is super useful for automated testing.

For example, I want to run all tests. I enable all configs,cool. But
how do I know that the desired code under test actually ran?

Another example, okay I ran my tests and some of them failed. All my
test debug information is all mixed together and there is a lot of
noise present, also I am afraid that something else not part of the
test influenced the test or code under test behavior. So I want to
rerun the failed tests and the code that they test and nothing else.

> > I'm not opposed to this, but
> > I wonder whether maintainers are willing to accept that stuff for just
> > that specific usecase.
>
> This is why we are discussing this on this thread.
>
> Since we have 'make localmodconfig' already upstream, and since this
> would help both users of that and backports I'd argue it makes sense
> upstream. Otherwise I find it that having upstream 'make localmodconfig'
> but not wanting to improve that problem space rather odd.
>
> > > However, folks producing embedded systems *do* / *should* have a lot of
> > > knowledge of their systems, and so the type of scheme you have devised
> > > seems sensible for it.
> >
> > Usually we have (unless we need to do reverse engineering :o). But it's
> > a pretty time-consuming task. Especially if the requirements change
> > several times in the development or lifetime of a specific product.
> >
> > For example "oh, we now need eth", "naah, we don't wanna use usb
> > anymore", "let's take a different SoM", ... not that have pretty
> > orthogonal sets of configs we need to maintain: hardware- and non-
> > hardware-related ones. And hardware-related ones can fall into different
> > categories like fixed-attached/onboard vs. hotpluggable ones.
> >
> > Recently I had a case where the customer requested xattr support, so
> > I had to enable general xattr support as well as per-filesystem.
> > Pretty simple, but having lots of those cases quickly sums up. One of
> > the reasons why I've written my own little config generator.
>
> I agree that the problem space you are dealing with should be made
> easier.
>
> > >> In embedded world, we often have scenarios where we want a really
> > >> minimal kernel, but need to enable/disable certain hi-level peripherals
> > >> in the middle of the project (eg. "oh, we also need ethernet, but we
> > >> wanna drop usb"). There we'll have to find out what actual chip is,
> > >> its corresponding driver, required subsystems, etc, and also kick off
> > >> everything we don't need anymore.
> > >
> > > Right. One *should* be able to tell some tool, hey, here is the list of
> > > my desirable .config options. Go and figure out what I need to make that
> > > work and give me a resulting .config. Its not easy.
> >
> > I think I've already got into a pretty usable state - at least for my
> > projects. For now only supports a few boards and limited set of
> > features, but patches are always welcomed :)
>
> A solution upstream would be better ;)
>
> > >> I've thought about implementing some actual dependency tracking
> > >> (at least recording the auto-enabled symbols), but didn't expect that
> > >> to become practically usable anytime soon,
> > >
> > > The ability to easily ask the kernel to enable the components needed
> > > for a respective config option *is* very useful but indeed not easy.
> >
> > Yes, it would need to understand things like conditional definitions
> > to deduce that certain things need to be enabled first, before certain
> > drivers become choosable.
> >
> > > This is not the only space where this problem exists. Similar problem
> > > exists for distribution packages, and dependencies. Challenges have
> > > been made for proper research towards these problems, and such research
> > > has lead distributions to opt to enable some of these algorithms.
> >
> > The problem w/ dependencies is that there can be different types of
> > dependencies, as well as different types of software objects. Just
> > solving the expressions is only a part of the problem.
> >
> > > This begs the question if we could learn from similar efforts on Linux
> > > for these sorts of questions. One possibility here is to evaluate the
> > > prospect of using a SAT solver with Minimally Unsatisfiable Subformulas
> > > (MUSes) support, which should be be able to address thir problem. This
> > > prospect is ongoing and currrent R&D is active, for details refer to:
> > >
> > > https://kernelnewbies.org/KernelProjects/kconfig-sat
> >
> > Good tip, I'll have a look at it.
> >
> > > It certainly can be useful for components, ie, not hardware. But for
> > > hardware a one-to-one mapping of one driver to one config would be of
> > > much more use.
> >
> > Unfortunately, we don't have this 1:1 mapping. Often drivers support
> > different sets of devices, depending on other factors, sometimes sub-
> > options (eg. different hw versions), sometimes depending on other
> > subsystems, sometimes arch-specific, etc, etc.
> >
> > I think we should work towards that, but I doubt we'd reach that goal
> > anytime soon, and begs the question whether it's really worth all the
> > effort required for that.
>
> Code speaks, and fortunately its being worked on.
>
> > > It would be wonderful if for instance kconfig
> > > supported a way to group a major set of components under *one* config
> > > symbol and then say: "I want this major component enabled" and then it'd
> > > go and enable all the defaults which would be required for it.
> >
> > Yes, thought about that, too. For example have syms for selecting whole
> > boards and features of them - a bit like this:
> >
> >   --> Preconfigure for specific boards
> >       --> board A
> >       --> board B
> >       ...
> >   --> Enable board features
> >       --> Ethernet port
> >       --> Display
> >           --> Touch panel
> >       --> Audio
> >       ....
> >
> > BUT: this would turn into maintenance hell, so I dropped that idea.

I wonder if that would work for the testing scenario? I don't think it
is unreasonable for a test owner to provide a defconfig that makes it
possible to run their test. We could then merge these together to
create a kconfig to run all desired tests. Doesn't address all the
issues I mentioned here, but it's a start.

> > > An example is if you
> > > wanted to enable PCI on a system which didn't support it. Because of
> > > this, it seems you'd want *all* desirable configs and let a piece of
> > > software figure out what you need / can enable. And.. this is precisely
> > > where the SAT solver with MUSes could help...
> >
> > Yes, but this piece of software first needs to know whether eg. PCI
> > is available on that HW. Oh, and things like PCI could be a dependency
> > as well as an feature on its own, depending on how you gonna use it.
> > (eg. if directly access from userland or VMs).
>
> Yes, but that is a separate problem. The way 'make localmodconfig'
> addresses this is by keeping as modules things which are already
> modules. And keeping things it found as =y as buit-in in your kernel as
> well.

Cheers!

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Enrico Weigelt, metux IT consult]

On 12.07.19 01:27, Brendan Higgins wrote:

> Enrico, want me to CC you on that> and we can continue this discussion there?
Yes. But I'd prefer having an own list for it - better for sorting and
archiving ;-)

> I wonder if that would work for the testing scenario? I don't think it> is unreasonable for a test owner to provide a defconfig that makes it>
possible to run their test. We could then merge these together to>
create a kconfig to run all desired tests. Doesn't address all the>
issues I mentioned here, but it's a start.
defconfig is a different thing - my idea (that I've dropped) was
actually introducing new config options per board (and sub options
for board features) which switch on all the neccessary things.

The defconfigs are nice thing for starting off with some board, but
they're basically examples, not production configurations. Yet another
point why I've decided to cope all of this in a separate tool.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287
--
To unsubscribe from this list: send the line "unsubscribe backports" in

Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Enrico Weigelt, metux IT consult]

On 12.07.19 01:27, Brendan Higgins wrote:

> Enrico, want me to CC you on that> and we can continue this discussion there?
Yes. But I'd prefer having an own list for it - better for sorting and
archiving ;-)

> I wonder if that would work for the testing scenario? I don't think it> is unreasonable for a test owner to provide a defconfig that makes it>
possible to run their test. We could then merge these together to>
create a kconfig to run all desired tests. Doesn't address all the>
issues I mentioned here, but it's a start.
defconfig is a different thing - my idea (that I've dropped) was
actually introducing new config options per board (and sub options
for board features) which switch on all the neccessary things.

The defconfigs are nice thing for starting off with some board, but
they're basically examples, not production configurations. Yet another
point why I've decided to cope all of this in a separate tool.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287