* [Cocci] [PATCH v2 0/10] Symbol namespaces - RFC [not found] <20180716122125.175792-1-maco@android.com> @ 2019-08-13 12:16 ` Matthias Maennich 2019-08-13 12:17 ` [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies Matthias Maennich [not found] ` <20190821114955.12788-1-maennich@google.com> 0 siblings, 2 replies; 9+ messages in thread From: Matthias Maennich @ 2019-08-13 12:16 UTC (permalink / raw) To: linux-kernel, maco Cc: kstewart, linux-m68k, oneukum, Mauro Carvalho Chehab, usb-storage, David Howells, Will Deacon, hpa, joel, sam, cocci, linux-arch, linux-scsi, Michael Ellerman, x86, lucas.de.marchi, mingo, geert, Johannes Weiner, stern, kernel-team, Dan Williams, Ingo Molnar, sspatil, arnd, linux-kbuild, jeyu, maennich, Jonathan Cameron, tglx, Adrian Reber, michal.lkml, Ard Biesheuvel, Richard Guy Briggs, gregkh, linux-usb, sboyd, Nicolas Ferre, Patrick Bellasi, maco, pombredanne, Andrew Morton, David S. Miller, linux-modules As of Linux 5.3-rc4, there are 31203 [1] exported symbols in the kernel. That is a growth of almost 1000 symbols since 4.17 (30206 [2]). There seems to be some consensus amongst kernel devs that the export surface is too large, and hard to reason about. Generally, these symbols fall in one of these categories: 1) Symbols actually meant for drivers 2) Symbols that are only exported because functionality is split over multiple modules, yet they really shouldn't be used by modules outside of their own subsystem 3) Symbols really only meant for in-tree use When module developers try to upstream their code, it regularly turns out that they are using exported symbols that they really shouldn't be using. This problem is even bigger for drivers that are currently out-of-tree, which may be using many symbols that they shouldn't be using, and that break when those symbols are removed or modified. This patch allows subsystem maintainers to partition their exported symbols into separate namespaces, and module authors to import such namespaces only when needed. This allows subsystem maintainers to more easily limit availability of these namespaced symbols to other parts of the kernel. It can also be used to partition the set of exported symbols for documentation purposes; for example, a set of symbols that is really only used for debugging could be in a "SUBSYSTEM_DEBUG" namespace. I continued the work mainly done by Martijn Coenen. In this v2 the following changes have been introduced compared to v1 of this series: - Rather than adding and evaluating separate sections __knsimport_NS, use modinfo tags to declare the namespaces a module introduces. Adjust modpost and the module loader accordingly. - Also add support for reading multiple modinfo values for the same tag to allow list-like access to modinfo tags. - The macros in export.h have been cleaned up to avoid redundancy in the macro parameters (ns, nspost, nspost2). - The introduction of relative references in the ksymtab entries caused a rework of the macros to accommodate that configuration as well. - Alignment of kernel_symbol in the ksymtab needed to be fixed to allow growing the kernel_symbol struct. - Modpost does now also append the namespace suffix to the symbol entries in Module.symvers. - The configuration option MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS allows relaxing the enforcement of properly declared namespace imports at module loading time. - Symbols can be collectively exported into a namespace by defining DEFAULT_SYMBOL_NAMESPACE in the corresponding Makefile. - The requirement for a very recent coccinelle spatch has been lifted by simplifying the script. - nsdeps does now ensures MODULE_IMPORT_NS statements are sorted when patching the module source files. - Some minor bugs have been addressed in nsdeps to allow it to work with modules that have more than one source file. - The RFC for the usb-storage symbols has been simplified by using DEFAULT_SYMBOL_NAMESPACE=USB_STORAGE rather than explicitly exporting each and every symbol into that new namespace. This patch series was developed against v5.3-rc4. [1] git grep "^EXPORT_SYMBOL\w*(" v5.3-rc4 | wc -l [2] git grep "^EXPORT_SYMBOL\w*(" v4.17 | wc -l Matthias Maennich (10): module: support reading multiple values per modinfo tag export: explicitly align struct kernel_symbol module: add support for symbol namespaces. modpost: add support for symbol namespaces module: add config option MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS export: allow definition default namespaces in Makefiles or sources modpost: add support for generating namespace dependencies scripts: Coccinelle script for namespace dependencies. usb-storage: remove single-use define for debugging RFC: usb-storage: export symbols in USB_STORAGE namespace MAINTAINERS | 5 + Makefile | 12 ++ arch/m68k/include/asm/export.h | 1 - drivers/usb/storage/Makefile | 2 + drivers/usb/storage/alauda.c | 1 + drivers/usb/storage/cypress_atacb.c | 1 + drivers/usb/storage/datafab.c | 1 + drivers/usb/storage/debug.h | 2 - drivers/usb/storage/ene_ub6250.c | 1 + drivers/usb/storage/freecom.c | 1 + drivers/usb/storage/isd200.c | 1 + drivers/usb/storage/jumpshot.c | 1 + drivers/usb/storage/karma.c | 1 + drivers/usb/storage/onetouch.c | 1 + drivers/usb/storage/realtek_cr.c | 1 + drivers/usb/storage/scsiglue.c | 2 +- drivers/usb/storage/sddr09.c | 1 + drivers/usb/storage/sddr55.c | 1 + drivers/usb/storage/shuttle_usbat.c | 1 + drivers/usb/storage/uas.c | 1 + include/asm-generic/export.h | 14 +- include/linux/export.h | 92 +++++++++++-- include/linux/module.h | 2 + init/Kconfig | 14 ++ kernel/module.c | 67 ++++++++- scripts/Makefile.modpost | 4 +- scripts/coccinelle/misc/add_namespace.cocci | 23 ++++ scripts/mod/modpost.c | 144 ++++++++++++++++++-- scripts/mod/modpost.h | 9 ++ scripts/nsdeps | 54 ++++++++ 30 files changed, 421 insertions(+), 40 deletions(-) create mode 100644 scripts/coccinelle/misc/add_namespace.cocci create mode 100644 scripts/nsdeps -- 2.23.0.rc1.153.gdeed80330f-goog _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
* [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies. 2019-08-13 12:16 ` [Cocci] [PATCH v2 0/10] Symbol namespaces - RFC Matthias Maennich @ 2019-08-13 12:17 ` Matthias Maennich 2019-08-13 12:31 ` Julia Lawall ` (3 more replies) [not found] ` <20190821114955.12788-1-maennich@google.com> 1 sibling, 4 replies; 9+ messages in thread From: Matthias Maennich @ 2019-08-13 12:17 UTC (permalink / raw) To: linux-kernel, maco Cc: kstewart, linux-m68k, oneukum, Mauro Carvalho Chehab, usb-storage, hpa, joel, sam, cocci, linux-arch, linux-scsi, x86, lucas.de.marchi, mingo, geert, stern, kernel-team, sspatil, arnd, linux-kbuild, Nicolas Palix, jeyu, maennich, Jonathan Cameron, tglx, michal.lkml, gregkh, linux-usb, sboyd, Nicolas Ferre, maco, pombredanne, David S. Miller, linux-modules A script that uses the '<module>.ns_deps' file generated by modpost to automatically add the required symbol namespace dependencies to each module. Usage: 1) Move some symbols to a namespace with EXPORT_SYMBOL_NS() or define DEFAULT_SYMBOL_NAMESPACE 2) Run 'make' (or 'make modules') and get warnings about modules not importing that namespace. 3) Run 'make nsdeps' to automatically add required import statements to said modules. This makes it easer for subsystem maintainers to introduce and maintain symbol namespaces into their codebase. Co-developed-by: Martijn Coenen <maco@android.com> Signed-off-by: Martijn Coenen <maco@android.com> Signed-off-by: Matthias Maennich <maennich@google.com> --- MAINTAINERS | 5 ++ Makefile | 12 +++++ scripts/Makefile.modpost | 4 +- scripts/coccinelle/misc/add_namespace.cocci | 23 +++++++++ scripts/nsdeps | 54 +++++++++++++++++++++ 5 files changed, 97 insertions(+), 1 deletion(-) create mode 100644 scripts/coccinelle/misc/add_namespace.cocci create mode 100644 scripts/nsdeps diff --git a/MAINTAINERS b/MAINTAINERS index e81e60bd7c26..aa169070a052 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -11414,6 +11414,11 @@ S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/wtarreau/nolibc.git F: tools/include/nolibc/ +NSDEPS +M: Matthias Maennich <maennich@google.com> +S: Maintained +F: scripts/nsdeps + NTB AMD DRIVER M: Shyam Sundar S K <Shyam-sundar.S-k@amd.com> L: linux-ntb@googlegroups.com diff --git a/Makefile b/Makefile index 1b23f95db176..c5c3356e133c 100644 --- a/Makefile +++ b/Makefile @@ -1500,6 +1500,9 @@ help: @echo ' headerdep - Detect inclusion cycles in headers' @echo ' coccicheck - Check with Coccinelle' @echo '' + @echo 'Tools:' + @echo ' nsdeps - Generate missing symbol namespace dependencies' + @echo '' @echo 'Kernel selftest:' @echo ' kselftest - Build and run kernel selftest (run as root)' @echo ' Build, install, and boot kernel before' @@ -1687,6 +1690,15 @@ quiet_cmd_tags = GEN $@ tags TAGS cscope gtags: FORCE $(call cmd,tags) +# Script to generate missing namespace dependencies +# --------------------------------------------------------------------------- + +PHONY += nsdeps + +nsdeps: + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost nsdeps + $(Q)$(CONFIG_SHELL) $(srctree)/scripts/$@ + # Scripts to check various things for consistency # --------------------------------------------------------------------------- diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost index 26e6574ecd08..743fe3a2e885 100644 --- a/scripts/Makefile.modpost +++ b/scripts/Makefile.modpost @@ -56,7 +56,8 @@ MODPOST = scripts/mod/modpost \ $(if $(KBUILD_EXTMOD),$(addprefix -e ,$(KBUILD_EXTRA_SYMBOLS))) \ $(if $(KBUILD_EXTMOD),-o $(modulesymfile)) \ $(if $(CONFIG_SECTION_MISMATCH_WARN_ONLY),,-E) \ - $(if $(KBUILD_MODPOST_WARN),-w) + $(if $(KBUILD_MODPOST_WARN),-w) \ + $(if $(filter nsdeps,$(MAKECMDGOALS)),-d) ifdef MODPOST_VMLINUX @@ -134,6 +135,7 @@ $(modules): %.ko :%.o %.mod.o FORCE targets += $(modules) +nsdeps: __modpost # Add FORCE to the prequisites of a target to force it to be always rebuilt. # --------------------------------------------------------------------------- diff --git a/scripts/coccinelle/misc/add_namespace.cocci b/scripts/coccinelle/misc/add_namespace.cocci new file mode 100644 index 000000000000..c832bb6445a8 --- /dev/null +++ b/scripts/coccinelle/misc/add_namespace.cocci @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0-only +// +/// Adds missing MODULE_IMPORT_NS statements to source files +/// +/// This script is usually called from scripts/nsdeps with -D ns=<namespace> to +/// add a missing namespace tag to a module source file. +/// + +@has_ns_import@ +declarer name MODULE_IMPORT_NS; +identifier virtual.ns; +@@ +MODULE_IMPORT_NS(ns); + +// Add missing imports, but only adjacent to a MODULE_LICENSE statement. +// That ensures we are adding it only to the main module source file. +@do_import depends on !has_ns_import@ +declarer name MODULE_LICENSE; +expression license; +identifier virtual.ns; +@@ +MODULE_LICENSE(license); ++ MODULE_IMPORT_NS(ns); diff --git a/scripts/nsdeps b/scripts/nsdeps new file mode 100644 index 000000000000..148db65a830f --- /dev/null +++ b/scripts/nsdeps @@ -0,0 +1,54 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Linux kernel symbol namespace import generator +# +# This script requires at least spatch +# version 1.0.4. +SPATCH_REQ_VERSION="1.0.4" + +DIR="$(dirname $(readlink -f $0))/.." +SPATCH="`which ${SPATCH:=spatch}`" +if [ ! -x "$SPATCH" ]; then + echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/' + exit 1 +fi + +SPATCH_REQ_VERSION_NUM=$(echo $SPATCH_REQ_VERSION | ${DIR}/scripts/ld-version.sh) +SPATCH_VERSION=$($SPATCH --version | head -1 | awk '{print $3}') +SPATCH_VERSION_NUM=$(echo $SPATCH_VERSION | ${DIR}/scripts/ld-version.sh) + +if [ "$SPATCH_VERSION_NUM" -lt "$SPATCH_REQ_VERSION_NUM" ] ; then + echo 'spatch needs to be version 1.06 or higher' + exit 1 +fi + +generate_deps_for_ns() { + $SPATCH --very-quiet --in-place --sp-file \ + $srctree/scripts/coccinelle/misc/add_namespace.cocci -D ns=$1 $2 +} + +generate_deps() { + local mod_file=`echo $@ | sed -e 's/\.ns_deps/\.mod/'` + local mod_name=`cat $mod_file | sed -n 1p | sed -e 's/\/[^.]*$//'` + local mod_source_files=`cat $mod_file | sed -n 2p | sed -e 's/\.o/\.c/g'` + for ns in `cat $@`; do + echo "Adding namespace $ns to module $mod_name (if needed)." + generate_deps_for_ns $ns $mod_source_files + # sort the imports + for source_file in $mod_source_files; do + sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmp + offset=$(wc -l ${source_file}.tmp | awk '{print $1;}') + cat $source_file | grep MODULE_IMPORT_NS | sort -u >> ${source_file}.tmp + tail -n +$((offset +1)) ${source_file} | grep -v MODULE_IMPORT_NS >> ${source_file}.tmp + if ! diff -q ${source_file} ${source_file}.tmp; then + mv ${source_file}.tmp ${source_file} + else + rm ${source_file}.tmp + fi + done + done +} + +for f in `find $srctree/.tmp_versions/ -name *.ns_deps`; do + generate_deps $f +done -- 2.23.0.rc1.153.gdeed80330f-goog _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies. 2019-08-13 12:17 ` [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies Matthias Maennich @ 2019-08-13 12:31 ` Julia Lawall 2019-08-13 12:44 ` Greg KH ` (2 subsequent siblings) 3 siblings, 0 replies; 9+ messages in thread From: Julia Lawall @ 2019-08-13 12:31 UTC (permalink / raw) To: Matthias Maennich Cc: kstewart, linux-m68k, oneukum, Mauro Carvalho Chehab, usb-storage, Nicolas Ferre, hpa, joel, sam, cocci, linux-arch, linux-scsi, x86, lucas.de.marchi, mingo, geert, stern, kernel-team, sspatil, arnd, linux-kbuild, jeyu, Nicolas Palix, Jonathan Cameron, tglx, maco, michal.lkml, gregkh, linux-usb, sboyd, linux-kernel, maco, pombredanne, David S. Miller, linux-modules On Tue, 13 Aug 2019, Matthias Maennich wrote: > A script that uses the '<module>.ns_deps' file generated by modpost to > automatically add the required symbol namespace dependencies to each > module. > > Usage: > 1) Move some symbols to a namespace with EXPORT_SYMBOL_NS() or define > DEFAULT_SYMBOL_NAMESPACE > 2) Run 'make' (or 'make modules') and get warnings about modules not > importing that namespace. > 3) Run 'make nsdeps' to automatically add required import statements > to said modules. > > This makes it easer for subsystem maintainers to introduce and maintain > symbol namespaces into their codebase. > > Co-developed-by: Martijn Coenen <maco@android.com> > Signed-off-by: Martijn Coenen <maco@android.com> > Signed-off-by: Matthias Maennich <maennich@google.com> Acked-by: Julia Lawall <julia.lawall@lip6.fr> > --- > MAINTAINERS | 5 ++ > Makefile | 12 +++++ > scripts/Makefile.modpost | 4 +- > scripts/coccinelle/misc/add_namespace.cocci | 23 +++++++++ > scripts/nsdeps | 54 +++++++++++++++++++++ > 5 files changed, 97 insertions(+), 1 deletion(-) > create mode 100644 scripts/coccinelle/misc/add_namespace.cocci > create mode 100644 scripts/nsdeps > > diff --git a/MAINTAINERS b/MAINTAINERS > index e81e60bd7c26..aa169070a052 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -11414,6 +11414,11 @@ S: Maintained > T: git git://git.kernel.org/pub/scm/linux/kernel/git/wtarreau/nolibc.git > F: tools/include/nolibc/ > > +NSDEPS > +M: Matthias Maennich <maennich@google.com> > +S: Maintained > +F: scripts/nsdeps > + > NTB AMD DRIVER > M: Shyam Sundar S K <Shyam-sundar.S-k@amd.com> > L: linux-ntb@googlegroups.com > diff --git a/Makefile b/Makefile > index 1b23f95db176..c5c3356e133c 100644 > --- a/Makefile > +++ b/Makefile > @@ -1500,6 +1500,9 @@ help: > @echo ' headerdep - Detect inclusion cycles in headers' > @echo ' coccicheck - Check with Coccinelle' > @echo '' > + @echo 'Tools:' > + @echo ' nsdeps - Generate missing symbol namespace dependencies' > + @echo '' > @echo 'Kernel selftest:' > @echo ' kselftest - Build and run kernel selftest (run as root)' > @echo ' Build, install, and boot kernel before' > @@ -1687,6 +1690,15 @@ quiet_cmd_tags = GEN $@ > tags TAGS cscope gtags: FORCE > $(call cmd,tags) > > +# Script to generate missing namespace dependencies > +# --------------------------------------------------------------------------- > + > +PHONY += nsdeps > + > +nsdeps: > + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost nsdeps > + $(Q)$(CONFIG_SHELL) $(srctree)/scripts/$@ > + > # Scripts to check various things for consistency > # --------------------------------------------------------------------------- > > diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost > index 26e6574ecd08..743fe3a2e885 100644 > --- a/scripts/Makefile.modpost > +++ b/scripts/Makefile.modpost > @@ -56,7 +56,8 @@ MODPOST = scripts/mod/modpost \ > $(if $(KBUILD_EXTMOD),$(addprefix -e ,$(KBUILD_EXTRA_SYMBOLS))) \ > $(if $(KBUILD_EXTMOD),-o $(modulesymfile)) \ > $(if $(CONFIG_SECTION_MISMATCH_WARN_ONLY),,-E) \ > - $(if $(KBUILD_MODPOST_WARN),-w) > + $(if $(KBUILD_MODPOST_WARN),-w) \ > + $(if $(filter nsdeps,$(MAKECMDGOALS)),-d) > > ifdef MODPOST_VMLINUX > > @@ -134,6 +135,7 @@ $(modules): %.ko :%.o %.mod.o FORCE > > targets += $(modules) > > +nsdeps: __modpost > > # Add FORCE to the prequisites of a target to force it to be always rebuilt. > # --------------------------------------------------------------------------- > diff --git a/scripts/coccinelle/misc/add_namespace.cocci b/scripts/coccinelle/misc/add_namespace.cocci > new file mode 100644 > index 000000000000..c832bb6445a8 > --- /dev/null > +++ b/scripts/coccinelle/misc/add_namespace.cocci > @@ -0,0 +1,23 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +// > +/// Adds missing MODULE_IMPORT_NS statements to source files > +/// > +/// This script is usually called from scripts/nsdeps with -D ns=<namespace> to > +/// add a missing namespace tag to a module source file. > +/// > + > +@has_ns_import@ > +declarer name MODULE_IMPORT_NS; > +identifier virtual.ns; > +@@ > +MODULE_IMPORT_NS(ns); > + > +// Add missing imports, but only adjacent to a MODULE_LICENSE statement. > +// That ensures we are adding it only to the main module source file. > +@do_import depends on !has_ns_import@ > +declarer name MODULE_LICENSE; > +expression license; > +identifier virtual.ns; > +@@ > +MODULE_LICENSE(license); > ++ MODULE_IMPORT_NS(ns); > diff --git a/scripts/nsdeps b/scripts/nsdeps > new file mode 100644 > index 000000000000..148db65a830f > --- /dev/null > +++ b/scripts/nsdeps > @@ -0,0 +1,54 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Linux kernel symbol namespace import generator > +# > +# This script requires at least spatch > +# version 1.0.4. > +SPATCH_REQ_VERSION="1.0.4" > + > +DIR="$(dirname $(readlink -f $0))/.." > +SPATCH="`which ${SPATCH:=spatch}`" > +if [ ! -x "$SPATCH" ]; then > + echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/' > + exit 1 > +fi > + > +SPATCH_REQ_VERSION_NUM=$(echo $SPATCH_REQ_VERSION | ${DIR}/scripts/ld-version.sh) > +SPATCH_VERSION=$($SPATCH --version | head -1 | awk '{print $3}') > +SPATCH_VERSION_NUM=$(echo $SPATCH_VERSION | ${DIR}/scripts/ld-version.sh) > + > +if [ "$SPATCH_VERSION_NUM" -lt "$SPATCH_REQ_VERSION_NUM" ] ; then > + echo 'spatch needs to be version 1.06 or higher' > + exit 1 > +fi > + > +generate_deps_for_ns() { > + $SPATCH --very-quiet --in-place --sp-file \ > + $srctree/scripts/coccinelle/misc/add_namespace.cocci -D ns=$1 $2 > +} > + > +generate_deps() { > + local mod_file=`echo $@ | sed -e 's/\.ns_deps/\.mod/'` > + local mod_name=`cat $mod_file | sed -n 1p | sed -e 's/\/[^.]*$//'` > + local mod_source_files=`cat $mod_file | sed -n 2p | sed -e 's/\.o/\.c/g'` > + for ns in `cat $@`; do > + echo "Adding namespace $ns to module $mod_name (if needed)." > + generate_deps_for_ns $ns $mod_source_files > + # sort the imports > + for source_file in $mod_source_files; do > + sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmp > + offset=$(wc -l ${source_file}.tmp | awk '{print $1;}') > + cat $source_file | grep MODULE_IMPORT_NS | sort -u >> ${source_file}.tmp > + tail -n +$((offset +1)) ${source_file} | grep -v MODULE_IMPORT_NS >> ${source_file}.tmp > + if ! diff -q ${source_file} ${source_file}.tmp; then > + mv ${source_file}.tmp ${source_file} > + else > + rm ${source_file}.tmp > + fi > + done > + done > +} > + > +for f in `find $srctree/.tmp_versions/ -name *.ns_deps`; do > + generate_deps $f > +done > -- > 2.23.0.rc1.153.gdeed80330f-goog > > _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies. 2019-08-13 12:17 ` [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies Matthias Maennich 2019-08-13 12:31 ` Julia Lawall @ 2019-08-13 12:44 ` Greg KH 2019-08-14 6:36 ` Himanshu Jha 2019-08-14 12:00 ` [Cocci] [v2 " Markus Elfring 3 siblings, 0 replies; 9+ messages in thread From: Greg KH @ 2019-08-13 12:44 UTC (permalink / raw) To: Matthias Maennich Cc: kstewart, linux-m68k, oneukum, Mauro Carvalho Chehab, usb-storage, Nicolas Ferre, hpa, joel, sam, cocci, linux-arch, linux-scsi, x86, lucas.de.marchi, mingo, geert, stern, kernel-team, sspatil, arnd, linux-kbuild, jeyu, Nicolas Palix, Jonathan Cameron, tglx, maco, michal.lkml, linux-usb, sboyd, linux-kernel, maco, pombredanne, David S. Miller, linux-modules On Tue, Aug 13, 2019 at 01:17:05PM +0100, Matthias Maennich wrote: > A script that uses the '<module>.ns_deps' file generated by modpost to > automatically add the required symbol namespace dependencies to each > module. > > Usage: > 1) Move some symbols to a namespace with EXPORT_SYMBOL_NS() or define > DEFAULT_SYMBOL_NAMESPACE > 2) Run 'make' (or 'make modules') and get warnings about modules not > importing that namespace. > 3) Run 'make nsdeps' to automatically add required import statements > to said modules. > > This makes it easer for subsystem maintainers to introduce and maintain > symbol namespaces into their codebase. > > Co-developed-by: Martijn Coenen <maco@android.com> > Signed-off-by: Martijn Coenen <maco@android.com> > Signed-off-by: Matthias Maennich <maennich@google.com> > --- I really can't express just how cool this patch is. I was amazed when I first saw it in action a long time ago, and still am. Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies. 2019-08-13 12:17 ` [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies Matthias Maennich 2019-08-13 12:31 ` Julia Lawall 2019-08-13 12:44 ` Greg KH @ 2019-08-14 6:36 ` Himanshu Jha 2019-08-14 8:03 ` Matthias Maennich 2019-08-14 12:00 ` [Cocci] [v2 " Markus Elfring 3 siblings, 1 reply; 9+ messages in thread From: Himanshu Jha @ 2019-08-14 6:36 UTC (permalink / raw) To: Matthias Maennich Cc: kstewart, linux-m68k, linux-usb, usb-storage, Nicolas Ferre, Mauro Carvalho Chehab, hpa, joel, sam, cocci, linux-arch, linux-scsi, x86, lucas.de.marchi, mingo, geert, stern, kernel-team, sspatil, arnd, linux-kbuild, pombredanne, Nicolas Palix, Jonathan Cameron, tglx, maco, michal.lkml, gregkh, oneukum, sboyd, linux-kernel, maco, jeyu, David S. Miller, linux-modules On Tue, Aug 13, 2019 at 01:17:05PM +0100, Matthias Maennich wrote: > A script that uses the '<module>.ns_deps' file generated by modpost to > automatically add the required symbol namespace dependencies to each > module. > > Usage: > 1) Move some symbols to a namespace with EXPORT_SYMBOL_NS() or define > DEFAULT_SYMBOL_NAMESPACE > 2) Run 'make' (or 'make modules') and get warnings about modules not > importing that namespace. > 3) Run 'make nsdeps' to automatically add required import statements > to said modules. > > This makes it easer for subsystem maintainers to introduce and maintain > symbol namespaces into their codebase. > > Co-developed-by: Martijn Coenen <maco@android.com> > Signed-off-by: Martijn Coenen <maco@android.com> > Signed-off-by: Matthias Maennich <maennich@google.com> > --- [] > MAINTAINERS | 5 ++ > Makefile | 12 +++++ > scripts/Makefile.modpost | 4 +- > scripts/coccinelle/misc/add_namespace.cocci | 23 +++++++++ > scripts/nsdeps | 54 +++++++++++++++++++++ > 5 files changed, 97 insertions(+), 1 deletion(-) > create mode 100644 scripts/coccinelle/misc/add_namespace.cocci > create mode 100644 scripts/nsdeps [] > +if [ "$SPATCH_VERSION_NUM" -lt "$SPATCH_REQ_VERSION_NUM" ] ; then > + echo 'spatch needs to be version 1.06 or higher' Nitpick: 1.0.6 > + exit 1 > +fi -- Himanshu Jha Undergraduate Student Department of Electronics & Communication Guru Tegh Bahadur Institute of Technology _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies. 2019-08-14 6:36 ` Himanshu Jha @ 2019-08-14 8:03 ` Matthias Maennich 0 siblings, 0 replies; 9+ messages in thread From: Matthias Maennich @ 2019-08-14 8:03 UTC (permalink / raw) To: Himanshu Jha Cc: kstewart, linux-m68k, linux-usb, usb-storage, Nicolas Ferre, Mauro Carvalho Chehab, hpa, joel, sam, cocci, linux-arch, linux-scsi, x86, lucas.de.marchi, mingo, geert, stern, kernel-team, sspatil, arnd, linux-kbuild, pombredanne, Nicolas Palix, Jonathan Cameron, tglx, maco, michal.lkml, gregkh, oneukum, linux-kernel, maco, jeyu, David S. Miller, linux-modules On Wed, Aug 14, 2019 at 12:06:11PM +0530, Himanshu Jha wrote: >On Tue, Aug 13, 2019 at 01:17:05PM +0100, Matthias Maennich wrote: >> A script that uses the '<module>.ns_deps' file generated by modpost to >> automatically add the required symbol namespace dependencies to each >> module. >> >> Usage: >> 1) Move some symbols to a namespace with EXPORT_SYMBOL_NS() or define >> DEFAULT_SYMBOL_NAMESPACE >> 2) Run 'make' (or 'make modules') and get warnings about modules not >> importing that namespace. >> 3) Run 'make nsdeps' to automatically add required import statements >> to said modules. >> >> This makes it easer for subsystem maintainers to introduce and maintain >> symbol namespaces into their codebase. >> >> Co-developed-by: Martijn Coenen <maco@android.com> >> Signed-off-by: Martijn Coenen <maco@android.com> >> Signed-off-by: Matthias Maennich <maennich@google.com> >> --- > >[] > >> MAINTAINERS | 5 ++ >> Makefile | 12 +++++ >> scripts/Makefile.modpost | 4 +- >> scripts/coccinelle/misc/add_namespace.cocci | 23 +++++++++ >> scripts/nsdeps | 54 +++++++++++++++++++++ >> 5 files changed, 97 insertions(+), 1 deletion(-) >> create mode 100644 scripts/coccinelle/misc/add_namespace.cocci >> create mode 100644 scripts/nsdeps > >[] > >> +if [ "$SPATCH_VERSION_NUM" -lt "$SPATCH_REQ_VERSION_NUM" ] ; then >> + echo 'spatch needs to be version 1.06 or higher' > >Nitpick: 1.0.6 > Good catch! Thanks! Actually it needs to be even 1.0.4, so I will just use $SPATCH_REQ_VERSION from above. >> + exit 1 >> +fi > > >-- >Himanshu Jha >Undergraduate Student >Department of Electronics & Communication >Guru Tegh Bahadur Institute of Technology _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Cocci] [v2 08/10] scripts: Coccinelle script for namespace dependencies 2019-08-13 12:17 ` [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies Matthias Maennich ` (2 preceding siblings ...) 2019-08-14 6:36 ` Himanshu Jha @ 2019-08-14 12:00 ` Markus Elfring 2019-08-14 12:20 ` Matthias Maennich 3 siblings, 1 reply; 9+ messages in thread From: Markus Elfring @ 2019-08-14 12:00 UTC (permalink / raw) To: Matthias Maennich, Greg Kroah-Hartman, Julia Lawall, Martijn Coenen, Himanshu Jha, cocci, kernel-janitors Cc: Kate Stewart, linux-m68k, Oliver Neukum, Mauro Carvalho Chehab, usb-storage, H. Peter Anvin, Joel Fernandes, Sam Ravnborg, linux-arch, linux-scsi, x86, Lucas De Marchi, Ingo Molnar, Alan Stern, Geert Uytterhoeven, kernel-team, , linux-kernel, Sandeep Patil, Arnd Bergmann, linux-kbuild, Philippe Ombredanne, Nicolas Palix, Jonathan Cameron, Thomas Gleixner, Michal Marek, linux-usb, Stephen Boyd, Nicolas Ferre, Martijn Coenen, Jessica Yu, David S. Miller, linux-modules +# This script requires at least spatch +# version 1.0.4. How do you think about to avoid the duplicate specification of this identification? Regards, Markus _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Cocci] [v2 08/10] scripts: Coccinelle script for namespace dependencies 2019-08-14 12:00 ` [Cocci] [v2 " Markus Elfring @ 2019-08-14 12:20 ` Matthias Maennich 0 siblings, 0 replies; 9+ messages in thread From: Matthias Maennich @ 2019-08-14 12:20 UTC (permalink / raw) To: Markus Elfring Cc: Kate Stewart, linux-m68k, Oliver Neukum, Mauro Carvalho Chehab, usb-storage, kernel-janitors, H. Peter Anvin, Joel Fernandes, Sam Ravnborg, cocci, linux-arch, linux-scsi, x86, Lucas De Marchi, Ingo Molnar, Alan Stern, Geert Uytterhoeven, kernel-team, Sandeep Patil, Arnd Bergmann, linux-kbuild, Philippe Ombredanne, Nicolas Palix, Jonathan Cameron, Thomas Gleixner, Martijn Coenen, Michal Marek, Greg Kroah-Hartman, linux-usb, Nicolas Ferre, Martijn Coenen, Jessica Yu, David S. Miller, linux-modules On Wed, Aug 14, 2019 at 02:00:23PM +0200, Markus Elfring wrote: >+# This script requires at least spatch >+# version 1.0.4. > >How do you think about to avoid the duplicate specification of this identification? > That is a good idea. I will reduce the patch to +# This script requires a minimum spatch version. +SPATCH_REQ_VERSION="1.0.4" and use $SPATCH_REQ_VERSION when diagnosing this issue further down. Cheers, Matthias _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <20190821114955.12788-1-maennich@google.com>]
[parent not found: <20190821114955.12788-9-maennich@google.com>]
* Re: [Cocci] [PATCH v3 08/11] scripts: Coccinelle script for namespace dependencies. [not found] ` <20190821114955.12788-9-maennich@google.com> @ 2019-08-29 12:13 ` Jessica Yu 0 siblings, 0 replies; 9+ messages in thread From: Jessica Yu @ 2019-08-29 12:13 UTC (permalink / raw) To: Matthias Maennich Cc: kstewart, linux-m68k, oneukum, Mauro Carvalho Chehab, usb-storage, Nicolas Ferre, hpa, joel, sam, cocci, linux-arch, linux-scsi, x86, lucas.de.marchi, mingo, geert, stern, kernel-team, sspatil, arnd, linux-kbuild, Nicolas Palix, tglx, maco, michal.lkml, gregkh, linux-usb, linux-kernel, maco, pombredanne, David S. Miller, linux-modules +++ Matthias Maennich [21/08/19 12:49 +0100]: >A script that uses the '<module>.ns_deps' files generated by modpost to >automatically add the required symbol namespace dependencies to each >module. > >Usage: >1) Move some symbols to a namespace with EXPORT_SYMBOL_NS() or define > DEFAULT_SYMBOL_NAMESPACE >2) Run 'make' (or 'make modules') and get warnings about modules not > importing that namespace. >3) Run 'make nsdeps' to automatically add required import statements > to said modules. > >This makes it easer for subsystem maintainers to introduce and maintain >symbol namespaces into their codebase. > >Co-developed-by: Martijn Coenen <maco@android.com> >Signed-off-by: Martijn Coenen <maco@android.com> >Acked-by: Julia Lawall <julia.lawall@lip6.fr> >Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> >Signed-off-by: Matthias Maennich <maennich@google.com> >--- > MAINTAINERS | 5 ++ > Makefile | 12 +++++ > scripts/Makefile.modpost | 4 +- > scripts/coccinelle/misc/add_namespace.cocci | 23 +++++++++ > scripts/nsdeps | 56 +++++++++++++++++++++ > 5 files changed, 99 insertions(+), 1 deletion(-) > create mode 100644 scripts/coccinelle/misc/add_namespace.cocci > create mode 100644 scripts/nsdeps > >diff --git a/MAINTAINERS b/MAINTAINERS >index 08176d64eed5..dd5b37b49a07 100644 >--- a/MAINTAINERS >+++ b/MAINTAINERS >@@ -11428,6 +11428,11 @@ S: Maintained > T: git git://git.kernel.org/pub/scm/linux/kernel/git/wtarreau/nolibc.git > F: tools/include/nolibc/ > >+NSDEPS >+M: Matthias Maennich <maennich@google.com> >+S: Maintained >+F: scripts/nsdeps >+ > NTB AMD DRIVER > M: Shyam Sundar S K <Shyam-sundar.S-k@amd.com> > L: linux-ntb@googlegroups.com >diff --git a/Makefile b/Makefile >index a89870188c09..40311f583ee1 100644 >--- a/Makefile >+++ b/Makefile >@@ -1500,6 +1500,9 @@ help: > @echo ' headerdep - Detect inclusion cycles in headers' > @echo ' coccicheck - Check with Coccinelle' > @echo '' >+ @echo 'Tools:' >+ @echo ' nsdeps - Generate missing symbol namespace dependencies' >+ @echo '' > @echo 'Kernel selftest:' > @echo ' kselftest - Build and run kernel selftest (run as root)' > @echo ' Build, install, and boot kernel before' >@@ -1687,6 +1690,15 @@ quiet_cmd_tags = GEN $@ > tags TAGS cscope gtags: FORCE > $(call cmd,tags) > >+# Script to generate missing namespace dependencies >+# --------------------------------------------------------------------------- >+ >+PHONY += nsdeps >+ >+nsdeps: >+ $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost nsdeps >+ $(Q)$(CONFIG_SHELL) $(srctree)/scripts/$@ >+ > # Scripts to check various things for consistency > # --------------------------------------------------------------------------- > >diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost >index 26e6574ecd08..743fe3a2e885 100644 >--- a/scripts/Makefile.modpost >+++ b/scripts/Makefile.modpost >@@ -56,7 +56,8 @@ MODPOST = scripts/mod/modpost \ > $(if $(KBUILD_EXTMOD),$(addprefix -e ,$(KBUILD_EXTRA_SYMBOLS))) \ > $(if $(KBUILD_EXTMOD),-o $(modulesymfile)) \ > $(if $(CONFIG_SECTION_MISMATCH_WARN_ONLY),,-E) \ >- $(if $(KBUILD_MODPOST_WARN),-w) >+ $(if $(KBUILD_MODPOST_WARN),-w) \ >+ $(if $(filter nsdeps,$(MAKECMDGOALS)),-d) > > ifdef MODPOST_VMLINUX > >@@ -134,6 +135,7 @@ $(modules): %.ko :%.o %.mod.o FORCE > > targets += $(modules) > >+nsdeps: __modpost > > # Add FORCE to the prequisites of a target to force it to be always rebuilt. > # --------------------------------------------------------------------------- >diff --git a/scripts/coccinelle/misc/add_namespace.cocci b/scripts/coccinelle/misc/add_namespace.cocci >new file mode 100644 >index 000000000000..c832bb6445a8 >--- /dev/null >+++ b/scripts/coccinelle/misc/add_namespace.cocci >@@ -0,0 +1,23 @@ >+// SPDX-License-Identifier: GPL-2.0-only >+// >+/// Adds missing MODULE_IMPORT_NS statements to source files >+/// >+/// This script is usually called from scripts/nsdeps with -D ns=<namespace> to >+/// add a missing namespace tag to a module source file. >+/// >+ >+@has_ns_import@ >+declarer name MODULE_IMPORT_NS; >+identifier virtual.ns; >+@@ >+MODULE_IMPORT_NS(ns); >+ >+// Add missing imports, but only adjacent to a MODULE_LICENSE statement. >+// That ensures we are adding it only to the main module source file. >+@do_import depends on !has_ns_import@ >+declarer name MODULE_LICENSE; >+expression license; >+identifier virtual.ns; >+@@ >+MODULE_LICENSE(license); >++ MODULE_IMPORT_NS(ns); >diff --git a/scripts/nsdeps b/scripts/nsdeps >new file mode 100644 >index 000000000000..3b5995a61e65 >--- /dev/null >+++ b/scripts/nsdeps >@@ -0,0 +1,56 @@ >+#!/bin/bash >+# SPDX-License-Identifier: GPL-2.0 >+# Linux kernel symbol namespace import generator >+# >+# This script requires a minimum spatch version. >+SPATCH_REQ_VERSION="1.0.4" >+ >+DIR="$(dirname $(readlink -f $0))/.." >+SPATCH="`which ${SPATCH:=spatch}`" >+if [ ! -x "$SPATCH" ]; then >+ echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/' >+ exit 1 >+fi >+ >+SPATCH_REQ_VERSION_NUM=$(echo $SPATCH_REQ_VERSION | ${DIR}/scripts/ld-version.sh) >+SPATCH_VERSION=$($SPATCH --version | head -1 | awk '{print $3}') >+SPATCH_VERSION_NUM=$(echo $SPATCH_VERSION | ${DIR}/scripts/ld-version.sh) >+ >+if [ "$SPATCH_VERSION_NUM" -lt "$SPATCH_REQ_VERSION_NUM" ] ; then >+ echo "spatch needs to be version $SPATCH_REQ_VERSION or higher" >+ exit 1 >+fi >+ >+generate_deps_for_ns() { >+ $SPATCH --very-quiet --in-place --sp-file \ >+ $srctree/scripts/coccinelle/misc/add_namespace.cocci -D ns=$1 $2 >+} >+ >+generate_deps() { >+ local mod_name=`basename $@ .ko` >+ local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` >+ local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` >+ if [ ! -f "$ns_deps_file" ]; then return; fi >+ local mod_source_files=`cat $mod_file | sed -n 1p | sed -e 's/\.o/\.c/g'` >+ for ns in `cat $ns_deps_file`; do >+ echo "Adding namespace $ns to module $mod_name (if needed)." >+ generate_deps_for_ns $ns $mod_source_files >+ # sort the imports >+ for source_file in $mod_source_files; do >+ sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmp >+ offset=$(wc -l ${source_file}.tmp | awk '{print $1;}') >+ cat $source_file | grep MODULE_IMPORT_NS | sort -u >> ${source_file}.tmp >+ tail -n +$((offset +1)) ${source_file} | grep -v MODULE_IMPORT_NS >> ${source_file}.tmp >+ if ! diff -q ${source_file} ${source_file}.tmp; then >+ mv ${source_file}.tmp ${source_file} >+ else >+ rm ${source_file}.tmp >+ fi >+ done >+ done >+} >+ >+for f in `cat $srctree/modules.order`; do >+ generate_deps $f >+done Hi Matthias! I normally build outside of the source tree (make O=..) and I think that choked up the nsdeps script a bit. For example when I run 'make nsdeps O=/tmp/linux' I get: cat: /home/jeyu/linux/modules.order: No such file or directory I just changed $srctree/modules.order to $objtree/modules.order and that fixed it for me. Also, I had to prefix $source_file in the script with $srctree so that it'd find the right file to modify. Thanks! Jessica _______________________________________________ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-09-05 11:56 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <20180716122125.175792-1-maco@android.com> 2019-08-13 12:16 ` [Cocci] [PATCH v2 0/10] Symbol namespaces - RFC Matthias Maennich 2019-08-13 12:17 ` [Cocci] [PATCH v2 08/10] scripts: Coccinelle script for namespace dependencies Matthias Maennich 2019-08-13 12:31 ` Julia Lawall 2019-08-13 12:44 ` Greg KH 2019-08-14 6:36 ` Himanshu Jha 2019-08-14 8:03 ` Matthias Maennich 2019-08-14 12:00 ` [Cocci] [v2 " Markus Elfring 2019-08-14 12:20 ` Matthias Maennich [not found] ` <20190821114955.12788-1-maennich@google.com> [not found] ` <20190821114955.12788-9-maennich@google.com> 2019-08-29 12:13 ` [Cocci] [PATCH v3 08/11] " Jessica Yu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).