Linux-Modules Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH kmod] Do not check for undefined symbols when building the Python modules
@ 2019-10-24 17:47 Thomas Petazzoni
  2019-10-25 22:07 ` Lucas De Marchi
  0 siblings, 1 reply; 2+ messages in thread
From: Thomas Petazzoni @ 2019-10-24 17:47 UTC (permalink / raw)
  To: linux-modules; +Cc: Thomas Petazzoni, Victor Stinner

kmod's configure.ac uses the -Wl,--no-undefined linker flag to verify
at link time that all symbols of shared libraries are available, and
that there are no undefined symbols.

This make perfect sense for regular shared libraries. However, for
Python extensions, which will be dlopen()ed inside the Python
interpreter, it makes less sense.

Since Python 3.8, there is a change in python-config script and
Python's pkg-config file: it no longer links Python extensions with
the libpython library. See
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
which states:

  On the other hand, pkg-config python3.8 --libs no longer contains
  -lpython3.8. C extensions must not be linked to libpython (except on
  Android and Cygwin, whose cases are handled by the script); this
  change is backward incompatible on purpose. (Contributed by Victor
  Stinner in bpo-36721.)

So, when linking the kmod Python extensions, it currently fails with
numerous unresolved symbols, that were previously provided by
libpython:

/home/test/autobuild/run/instance-3/output-1/host/opt/ext-toolchain/bin/../lib/gcc/powerpc64-buildroot-linux-gnu/7.4.0/../../../../powerpc64-buildroot-linux-gnu/bin/ld: libkmod/python/kmod/.libs/list_la-list.o: in function `__Pyx_PyObject_GetAttrStr':
list.c:(.text.__Pyx_PyObject_GetAttrStr+0x48): undefined reference to `PyObject_GetAttr'
/home/test/autobuild/run/instance-3/output-1/host/opt/ext-toolchain/bin/../lib/gcc/powerpc64-buildroot-linux-gnu/7.4.0/../../../../powerpc64-buildroot-linux-gnu/bin/ld: libkmod/python/kmod/.libs/list_la-list.o: in function `__pyx_tp_dealloc_4kmod_4list_ModListItem':
list.c:(.text.__pyx_tp_dealloc_4kmod_4list_ModListItem+0x78): undefined reference to `PyObject_CallFinalizerFromDealloc'
/home/test/autobuild/run/instance-3/output-1/host/opt/ext-toolchain/bin/../lib/gcc/powerpc64-buildroot-linux-gnu/7.4.0/../../../../powerpc64-buildroot-linux-gnu/bin/ld: libkmod/python/kmod/.libs/list_la-list.o: in function `__pyx_tp_dealloc_4kmod_4list_ModList':
list.c:(.text.__pyx_tp_dealloc_4kmod_4list_ModList+0x30): undefined reference to `PyErr_Fetch'

[Complete log at http://autobuild.buildroot.net/results/79a/79a5a0398723e8cfea0d0aa3dec5f7649aee4c63/build-end.log]

Linking with libpython is no longer recommended: those symbols should
remain unresolved in the Python extensions, as they wil be properly
resolved when the Python extension gets loaded into the Python
interpreter.

Since we want to keep -Wl,--no-undefined globally in kmod, we leave
the configure.ac file unchanged, and instead, specifically in the
LDFLAGS used to build the Python extensions, we override
-Wl,--no-undefined with -Wl,-z,undefs. Ideally, -Wl,--no-undefined is
the same as -Wl,-z,defs, and the effect of these options can be
canceled on the linker command line by a following -Wl,-z,undefs (see
the ld man page for details).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Victor Stinner <victor.stinner@gmail.com>
---
 Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index c5c2f06..8e9c90d 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -173,7 +173,7 @@ CPYTHON_MODULE_CFLAGS = \
 	$(AM_CFLAGS) -DCPYTHON_COMPILING_IN_PYPY=0 \
 	$(PYTHON_NOWARN) $(PYTHON_CFLAGS) \
 	-fvisibility=default
-CPYTHON_MODULE_LDFLAGS = $(AM_LDFLAGS) -module -avoid-version -shared
+CPYTHON_MODULE_LDFLAGS = $(AM_LDFLAGS) -module -avoid-version -shared -Wl,-z,undefs
 
 if BUILD_PYTHON
 pkgpyexec_LTLIBRARIES = \
-- 
2.21.0


^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [PATCH kmod] Do not check for undefined symbols when building the Python modules
  2019-10-24 17:47 [PATCH kmod] Do not check for undefined symbols when building the Python modules Thomas Petazzoni
@ 2019-10-25 22:07 ` Lucas De Marchi
  0 siblings, 0 replies; 2+ messages in thread
From: Lucas De Marchi @ 2019-10-25 22:07 UTC (permalink / raw)
  To: Thomas Petazzoni; +Cc: linux-modules, Victor Stinner

On Thu, Oct 24, 2019 at 07:47:10PM +0200, Thomas Petazzoni wrote:
>kmod's configure.ac uses the -Wl,--no-undefined linker flag to verify
>at link time that all symbols of shared libraries are available, and
>that there are no undefined symbols.
>
>This make perfect sense for regular shared libraries. However, for
>Python extensions, which will be dlopen()ed inside the Python
>interpreter, it makes less sense.
>
>Since Python 3.8, there is a change in python-config script and
>Python's pkg-config file: it no longer links Python extensions with
>the libpython library. See
>https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
>which states:
>
>  On the other hand, pkg-config python3.8 --libs no longer contains
>  -lpython3.8. C extensions must not be linked to libpython (except on
>  Android and Cygwin, whose cases are handled by the script); this
>  change is backward incompatible on purpose. (Contributed by Victor
>  Stinner in bpo-36721.)
>
>So, when linking the kmod Python extensions, it currently fails with
>numerous unresolved symbols, that were previously provided by
>libpython:
>
>/home/test/autobuild/run/instance-3/output-1/host/opt/ext-toolchain/bin/../lib/gcc/powerpc64-buildroot-linux-gnu/7.4.0/../../../../powerpc64-buildroot-linux-gnu/bin/ld: libkmod/python/kmod/.libs/list_la-list.o: in function `__Pyx_PyObject_GetAttrStr':
>list.c:(.text.__Pyx_PyObject_GetAttrStr+0x48): undefined reference to `PyObject_GetAttr'
>/home/test/autobuild/run/instance-3/output-1/host/opt/ext-toolchain/bin/../lib/gcc/powerpc64-buildroot-linux-gnu/7.4.0/../../../../powerpc64-buildroot-linux-gnu/bin/ld: libkmod/python/kmod/.libs/list_la-list.o: in function `__pyx_tp_dealloc_4kmod_4list_ModListItem':
>list.c:(.text.__pyx_tp_dealloc_4kmod_4list_ModListItem+0x78): undefined reference to `PyObject_CallFinalizerFromDealloc'
>/home/test/autobuild/run/instance-3/output-1/host/opt/ext-toolchain/bin/../lib/gcc/powerpc64-buildroot-linux-gnu/7.4.0/../../../../powerpc64-buildroot-linux-gnu/bin/ld: libkmod/python/kmod/.libs/list_la-list.o: in function `__pyx_tp_dealloc_4kmod_4list_ModList':
>list.c:(.text.__pyx_tp_dealloc_4kmod_4list_ModList+0x30): undefined reference to `PyErr_Fetch'
>
>[Complete log at http://autobuild.buildroot.net/results/79a/79a5a0398723e8cfea0d0aa3dec5f7649aee4c63/build-end.log]
>
>Linking with libpython is no longer recommended: those symbols should
>remain unresolved in the Python extensions, as they wil be properly
>resolved when the Python extension gets loaded into the Python
>interpreter.
>
>Since we want to keep -Wl,--no-undefined globally in kmod, we leave
>the configure.ac file unchanged, and instead, specifically in the
>LDFLAGS used to build the Python extensions, we override
>-Wl,--no-undefined with -Wl,-z,undefs. Ideally, -Wl,--no-undefined is
>the same as -Wl,-z,defs, and the effect of these options can be
>canceled on the linker command line by a following -Wl,-z,undefs (see
>the ld man page for details).
>
>Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>Cc: Victor Stinner <victor.stinner@gmail.com>

applied, thanks

Lucas De Marchi

>---
> Makefile.am | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>diff --git a/Makefile.am b/Makefile.am
>index c5c2f06..8e9c90d 100644
>--- a/Makefile.am
>+++ b/Makefile.am
>@@ -173,7 +173,7 @@ CPYTHON_MODULE_CFLAGS = \
> 	$(AM_CFLAGS) -DCPYTHON_COMPILING_IN_PYPY=0 \
> 	$(PYTHON_NOWARN) $(PYTHON_CFLAGS) \
> 	-fvisibility=default
>-CPYTHON_MODULE_LDFLAGS = $(AM_LDFLAGS) -module -avoid-version -shared
>+CPYTHON_MODULE_LDFLAGS = $(AM_LDFLAGS) -module -avoid-version -shared -Wl,-z,undefs
>
> if BUILD_PYTHON
> pkgpyexec_LTLIBRARIES = \
>-- 
>2.21.0
>

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, back to index

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-24 17:47 [PATCH kmod] Do not check for undefined symbols when building the Python modules Thomas Petazzoni
2019-10-25 22:07 ` Lucas De Marchi

Linux-Modules Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-modules/0 linux-modules/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-modules linux-modules/ https://lore.kernel.org/linux-modules \
		linux-modules@vger.kernel.org
	public-inbox-index linux-modules

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-modules


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git