[PATCH 4/4] KBUILD: export-symbol usage report generator

[PATCH 4/4] KBUILD: export-symbol usage report generator

[Ram Pai]

From: Ram Pai <linuxram@us.ibm.com>

The following patch provides the ability to generate a report of
     (1) All the exported symbols and their in-kernel-module usage count
     (2) For each module, lists the modules and their exported symbols, on
                which it depends.

The report generation is integrated in the build process.
'make exportcheck' prints out the report.
'make exportcheck EXPORTFILE=Documentation/export_report.txt'
        generates the report in the file Documentation/export_report.txt

NOTE: the same report can also be generated by executing
	perl scripts/export_report

Signed-off-by: Ram Pai <linuxram@us.ibm.com>

 Makefile                 |   16 ++++-
 scripts/Makefile.modpost |    7 ++
 scripts/export_report.pl |  148 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 170 insertions(+), 1 deletion(-)

Index: linux-2.6.17-rc3/scripts/Makefile.modpost
===================================================================
--- linux-2.6.17-rc3.orig/scripts/Makefile.modpost
+++ linux-2.6.17-rc3/scripts/Makefile.modpost
@@ -60,9 +60,16 @@ quiet_cmd_modpost = MODPOST
 	$(if $(KBUILD_EXTMOD),-o $(modulesymfile)) \
 	$(filter-out FORCE,$^)
 
+quiet_cmd_importfile = IMPORT_EXTRACT
+      cmd_importfile =  perl $(objtree)/scripts/export_report.pl  \
+	-k $(kernelsymfile) \
+	$(if $(KBUILD_EXPORT_REPORT:1=),-o $(KBUILD_EXPORT_REPORT),) \
+	-f $(patsubst %.o,%.mod.c,$(filter-out vmlinux FORCE, $^))
+
 PHONY += __modpost
 __modpost: $(wildcard vmlinux) $(modules:.ko=.o) FORCE
 	$(call cmd,modpost)
+	$(if $(KBUILD_EXPORT_REPORT), $(call cmd,importfile))
 
 # Declare generated files as targets for modpost
 $(symverfile):         __modpost ;
Index: linux-2.6.17-rc3/scripts/export_report.pl
===================================================================
--- /dev/null
+++ linux-2.6.17-rc3/scripts/export_report.pl
@@ -0,0 +1,148 @@
+#!/usr/bin/perl -w
+#
+# (C) Copyright IBM Corporation 2006.
+#	Released under GPL v2.
+#	Author : Ram Pai (linuxram@us.ibm.com)
+#
+# Usage: export_report.pl -k Module.symvers [-o report_file ] *.mod.c
+#
+
+use Getopt::Std;
+use strict;
+
+sub numerically {
+	my $no1 = (split /\s+/, $a)[1];
+	my $no2 = (split /\s+/, $b)[1];
+	return $no1 <=> $no2;
+}
+
+sub alphabetically {
+	my ($module1, $value1) = @{$a};
+	my ($module2, $value2) = @{$b};
+	return $value1 <=> $value2 || $module2 cmp $module1;
+}
+
+sub print_depends_on {
+	my ($href) = @_;
+	print "\n";
+	while (my ($mod, $list) = each %$href) {
+		print "\t$mod:\n";
+		foreach my $sym (sort numerically @{$list}) {
+			my ($symbol, $no) = split /\s+/, $sym;
+			printf("\t\t%-25s\t%-25d\n", $symbol, $no);
+		}
+		print "\n";
+	}
+	print "\n";
+	print "~"x80 , "\n";
+}
+
+sub usage {
+        die "Usage: @_ -h -k Module.symvers  [ -o outputfile ]  -f  all the .mod.c files \n";
+}
+
+sub collectcfiles {
+        my @file = `cat .tmp_versions/*.mod | grep '.*\.ko\$'`;
+        @file = grep {s/\.ko/.mod.c/} @file;
+        return @file;
+}
+
+my (%SYMBOL, %MODULE, %opt, @allcfiles);
+
+if (not getopts('hk:o:f',\%opt) or defined $opt{'h'}) {
+        usage($0);
+}
+
+if (defined $opt{'f'}) {
+	@allcfiles = @ARGV;
+} else {
+	@allcfiles = collectcfiles();
+}
+
+if (not defined $opt{'k'}) {
+	$opt{'k'} = "Module.symvers";
+}
+
+unless (open(MODULE_SYMVERS, $opt{'k'})) {
+	die "Sorry, cannot open $opt{'k'}: $!\n";
+}
+
+
+if (defined $opt{'o'}) {
+	unless (open(OUTPUT_HANDLE, ">$opt{'o'}")) {
+		die "Sorry, cannot open $opt{'o'} $!\n";
+	}
+	select OUTPUT_HANDLE;
+}
+#
+# collect all the symbols and their attributes from the
+# Module.symvers file
+#
+while ( <MODULE_SYMVERS> ) {
+	chomp;
+	my (undef, $symbol, $module, $gpl) = split;
+	$SYMBOL { $symbol } =  [ $module , "0" , $symbol, $gpl];
+}
+close(MODULE_SYMVERS);
+
+#
+# collect the usage count of each symbol.
+#
+foreach my $thismod (@allcfiles) {
+	unless (open(MODULE_MODULE, $thismod)) {
+		print "Sorry, cannot open $thismod: $!\n";
+		next;
+	}
+	while ( <MODULE_MODULE> ) {
+		chomp;
+		if ( $_ !~ /0x[0-9a-f]{7,8},/ ) {
+			next;
+		}
+		my $sym = (split /([,"])/,)[4];
+		my ($module, $value, $symbol, $gpl) = @{$SYMBOL{$sym}};
+		$SYMBOL{ $sym } =  [ $module, $value+1, $symbol, $gpl];
+		push(@{$MODULE{$thismod}} , $sym);
+	}
+	close(MODULE_MODULE);
+}
+
+print "\tTHIS FILE REPORTS THE USAGE PATTERNS OF EXPORTED SYMBOLS BY IN_TREE\n";
+print "\t\t\t\tMODULES\n";
+printf("%s\n\n\n","x"x80);
+printf("\t\t\t\tINDEX\n\n\n");
+printf("SECTION 1: USAGE COUNTS OF ALL EXPORTED SYMBOLS\n");
+printf("SECTION 2: LIST OF MODULES AND THE EXPORTED SYMBOLS THEY USE\n");
+printf("%s\n\n\n","x"x80);
+printf("SECTION 1:\tTHE EXPORTED SYMBOLS AND THEIR USAGE COUNT\n\n");
+printf("%-25s\t%-25s\t%-5s\t%-25s\n", "SYMBOL", "MODULE", "USAGE COUNT",
+	"EXPORT TYPE");
+
+#
+# print the list of unused exported symbols
+#
+foreach my $list (sort alphabetically values(%SYMBOL)) {
+	my ($module, $value, $symbol, $gpl) = @{$list};
+	printf("%-25s\t%-25s\t%-10s\t", $symbol, $module, $value);
+	if (defined $gpl) {
+		printf("%-25s\n",$gpl);
+	} else {
+		printf("\n");
+	}
+}
+printf("%s\n\n\n","x"x80);
+
+printf("SECTION 2:\n\tThis section reports export-symbol-usage of in-kernel
+modules. Each module lists the modules, and the symbols from that module that
+it uses.  Each listed symbol reports the number of modules using it\n");
+
+print "~"x80 , "\n";
+while (my ($thismod, $list) = each %MODULE) {
+	my %depends;
+	$thismod =~ s/\.mod\.c/.ko/;
+	print "\t\t\t$thismod\n";
+	foreach my $symbol (@{$list}) {
+		my ($module, $value, undef, $gpl) = @{$SYMBOL{$symbol}};
+		push (@{$depends{"$module"}}, "$symbol $value");
+	}
+	print_depends_on(\%depends);
+}
Index: linux-2.6.17-rc3/Makefile
===================================================================
--- linux-2.6.17-rc3.orig/Makefile
+++ linux-2.6.17-rc3/Makefile
@@ -207,7 +207,7 @@ endif
 #	in addition to whatever we do anyway.
 #	Just "make" or "make all" shall build modules as well
 
-ifneq ($(filter all _all modules,$(MAKECMDGOALS)),)
+ifneq ($(filter all _all modules exportcheck,$(MAKECMDGOALS)),)
   KBUILD_MODULES := 1
 endif
 
@@ -1029,6 +1029,8 @@ help:
 	@echo  '  cscope	  - Generate cscope index'
 	@echo  '  kernelrelease	  - Output the release version string'
 	@echo  '  kernelversion	  - Output the version stored in Makefile'
+	@echo  '  exportcheck [EXPORTFILE=file]  - Export symbol usage analysis '
+	@echo  '				on compiled kernel'
 	@echo  ''
 	@echo  'Static analysers'
 	@echo  '  checkstack      - Generate a list of stack hogs'
@@ -1258,6 +1260,18 @@ versioncheck:
 namespacecheck:
 	$(PERL) $(srctree)/scripts/namespace.pl
 
+ifeq ($(MAKECMDGOALS), exportcheck)
+  	KBUILD_EXPORT_REPORT := 1
+	ifdef EXPORTFILE
+	  ifeq ("$(origin EXPORTFILE)", "command line")
+  		KBUILD_EXPORT_REPORT = $(EXPORTFILE)
+	  endif
+	endif
+	export KBUILD_EXPORT_REPORT
+endif
+PHONY += exportcheck
+exportcheck: vmlinux modules
+
 endif #ifeq ($(config-targets),1)
 endif #ifeq ($(mixed-targets),1)
 

Re: [PATCH 4/4] KBUILD: export-symbol usage report generator

[Andi Kleen]

linuxram@us.ibm.com (Ram Pai) writes:

> From: Ram Pai <linuxram@us.ibm.com>
> 
> The following patch provides the ability to generate a report of
>      (1) All the exported symbols and their in-kernel-module usage count
>      (2) For each module, lists the modules and their exported symbols, on
>                 which it depends.

Very nice.

One thing I always wanted to see was a more focussed EXPORT_SYMBOL.

A lot of symbols are only exported for a single other module (e.g. most
of the networking exports are for IPv6) but are actually internal 
and shouldn't be messed with by other modules. It would be nice
if name spaces could be defined that say "this export is only for 
modules in this name space" and then e.g. have IPv6 be in the TCPINTERNALS
name space and nobody else.

I think adding something like this could clean up the bewildering
jungle of exports greatly.

_GPL is kind of like that already, but it is not fine grained enough.

-Andi

Re: [PATCH 4/4] KBUILD: export-symbol usage report generator

[Christoph Hellwig]

On Thu, May 11, 2006 at 01:43:27PM +0200, Andi Kleen wrote:
> linuxram@us.ibm.com (Ram Pai) writes:
> 
> > From: Ram Pai <linuxram@us.ibm.com>
> > 
> > The following patch provides the ability to generate a report of
> >      (1) All the exported symbols and their in-kernel-module usage count
> >      (2) For each module, lists the modules and their exported symbols, on
> >                 which it depends.
> 
> Very nice.
> 
> One thing I always wanted to see was a more focussed EXPORT_SYMBOL.
> 
> A lot of symbols are only exported for a single other module (e.g. most
> of the networking exports are for IPv6) but are actually internal 
> and shouldn't be messed with by other modules. It would be nice
> if name spaces could be defined that say "this export is only for 
> modules in this name space" and then e.g. have IPv6 be in the TCPINTERNALS
> name space and nobody else.
> 
> I think adding something like this could clean up the bewildering
> jungle of exports greatly.
> 
> _GPL is kind of like that already, but it is not fine grained enough.

I'd go even further and say every symbol should have a category.  These
catgories than could get labels such as mostly stable, public but volatile
or internal (the latter is what's oddly named _GPL) now.  That way a small
look at a driver can say if it's doing something wrong, e.g. if a network
driver imports symbols from anything but BASE, PCI and NETDEV something
is most likely wrong.

Re: [PATCH 4/4] KBUILD: export-symbol usage report generator

[Andi Kleen]

> I'd go even further and say every symbol should have a category. 

That would be a _lot_ of work. I was thinking more for some specialized
symbols first where the meaning is very clear. Also someone has to 
write the infrastructure first.

> These 
> catgories than could get labels such as mostly stable

I don't labels like "stable" etc. very much because it would give false
expectations to external users.

-Andi