[Buildroot] [PATCH 1/2] package/berkeleydb: add option for historic dbm interface

[Buildroot] [PATCH 1/2] package/berkeleydb: add option for historic dbm interface

[Bernd Kuhls]

Berkeley DB support in python depends on dbm, when missing python fails
to detect libdb:
https://github.com/python/cpython/blob/d2340ef25721b6a72d45d4508c672c4be38c67d3/configure.ac#L4002

quoting python configure log:
checking for libdb... no

quoting python config.log:
conftest.c:(.text.startup+0x8): undefined reference to `__db_ndbm_open'

This patch provides a configure option for the historic dbm interface
to be used by the python package.

Signed-off-by: Bernd Kuhls <bernd@kuhls.net>
---
 package/berkeleydb/Config.in     | 5 +++++
 package/berkeleydb/berkeleydb.mk | 1 +
 2 files changed, 6 insertions(+)

diff --git a/package/berkeleydb/Config.in b/package/berkeleydb/Config.in
index f32d13ab17..f8fbb5cc09 100644
--- a/package/berkeleydb/Config.in
+++ b/package/berkeleydb/Config.in
@@ -13,6 +13,11 @@ config BR2_PACKAGE_BERKELEYDB_COMPAT185
 	help
 	  Build and install DB 1.85 compatibility API.
 
+config BR2_PACKAGE_BERKELEYDB_DBM
+	bool "historic dbm interface"
+	help
+	  Enable the historic dbm interface.
+
 config BR2_PACKAGE_BERKELEYDB_TOOLS
 	bool "install tools"
 	help
diff --git a/package/berkeleydb/berkeleydb.mk b/package/berkeleydb/berkeleydb.mk
index 36b1894f34..1c121370db 100644
--- a/package/berkeleydb/berkeleydb.mk
+++ b/package/berkeleydb/berkeleydb.mk
@@ -39,6 +39,7 @@ define BERKELEYDB_CONFIGURE_CMDS
 		--disable-java \
 		--disable-tcl \
 		$(if $(BR2_PACKAGE_BERKELEYDB_COMPAT185),--enable-compat185,--disable-compat185) \
+		$(if $(BR2_PACKAGE_BERKELEYDB_DBM),--enable-dbm,--disable-dbm) \
 		$(SHARED_STATIC_LIBS_OPTS) \
 		--with-pic \
 		--enable-o_direct \
-- 
2.39.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] package/python3: berkeleydb support needs the dbm interface

[Bernd Kuhls]

Berkeley DB support in python depends on dbm, when missing python fails
to detect libdb:
https://github.com/python/cpython/blob/d2340ef25721b6a72d45d4508c672c4be38c67d3/configure.ac#L4002

quoting configure log:
checking for libdb... no

quoting python config.log:
conftest.c:(.text.startup+0x8): undefined reference to `__db_ndbm_open'

Only with this patch python builds a berkeleydb module:

  output/build/python3-3.11.4$ find -iname *dbm*.so
  ./build/lib.linux-x86_64-3.11/_dbm.cpython-311-x86_64-linux-gnu.so

since python 3.11 (added with buildroot commit
738500c296c8b1206f20e94ca3e7c5932a6a0486) changed the way it detects libdb:
https://github.com/python/cpython/commit/0a9f69539be27acf1cddf1b58d02a88d02e5008d

Signed-off-by: Bernd Kuhls <bernd@kuhls.net>
---
 package/python3/Config.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/python3/Config.in b/package/python3/Config.in
index 761c38c850..38f0580aa4 100644
--- a/package/python3/Config.in
+++ b/package/python3/Config.in
@@ -48,6 +48,7 @@ config BR2_PACKAGE_PYTHON3_2TO3
 config BR2_PACKAGE_PYTHON3_BERKELEYDB
 	bool "berkeleydb"
 	select BR2_PACKAGE_BERKELEYDB
+	select BR2_PACKAGE_BERKELEYDB_DBM
 	help
 	  berkeleydb module for Python3
 
-- 
2.39.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/berkeleydb: add option for historic dbm interface

[Thomas Petazzoni via buildroot]

Hello Bernd,

On Mon,  7 Aug 2023 19:45:14 +0200
Bernd Kuhls <bernd@kuhls.net> wrote:

> Berkeley DB support in python depends on dbm, when missing python fails
> to detect libdb:
> https://github.com/python/cpython/blob/d2340ef25721b6a72d45d4508c672c4be38c67d3/configure.ac#L4002
> 
> quoting python configure log:
> checking for libdb... no
> 
> quoting python config.log:
> conftest.c:(.text.startup+0x8): undefined reference to `__db_ndbm_open'
> 
> This patch provides a configure option for the historic dbm interface
> to be used by the python package.
> 
> Signed-off-by: Bernd Kuhls <bernd@kuhls.net>

Thanks for those patches. However, I don't quite understand how it was
working in Python 3.10.

I agree that in Python 3.11, they moved some detection logic from
setup.py to configure.ac. But the actual code of dbm module hasn't
changed. Back in Python 3.10, it was already calling dbm_open(), which
gets expanded by a macro from db.h into __db_ndbm_open(). And indeed in
Python 3.10, the dbm module already has references to this symbol:

$ readelf -aW output/target/usr/lib/python3.10/lib-dynload/_dbm.cpython-310-arm-linux-gnueabihf.so  | grep "UND.*__db"
    10: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_delete
    15: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_store
    23: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_nextkey
    25: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_clearerr
    40: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_fetch
    41: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_open
    44: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_firstkey
    48: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_error
    51: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __db_ndbm_close

And those symbols are not provided by libdb-5.3.so:

$ readelf -aW output/target/usr/lib/libdb-5.3.so | grep __db_ndbm
$

But, the dbm Python module works as expected:

# python
Python 3.10.8 (main, Aug  7 2023, 22:52:15) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbm
>>> dbm.open("/tmp/plop", "c")
<dbm.dumb._Database object at 0x767ccc28>

So I'm really confused. How does it work in Python 3.10 ? Did we
already need the historic dbm interface back then ?

Any idea?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/berkeleydb: add option for historic dbm interface

[Bernd Kuhls]

Hi,

Am Mon, 7 Aug 2023 23:14:38 +0200 schrieb Thomas Petazzoni via buildroot:

> I agree that in Python 3.11, they moved some detection logic from
> setup.py to configure.ac. But the actual code of dbm module hasn't
> changed.

the problem I am fixing is only about the detection during configure and 
unrelated to the code of the dbm module.

The detection logic in Python 3.11 tries to link a sample program 
containing "dbm_open()" to -ldb: https://github.com/python/cpython/blob/
d2340ef25721b6a72d45d4508c672c4be38c67d3/configure.ac#L4007

and fails, for whatever reason:
conftest.c:(.text.startup+0x8): undefined reference to `__db_ndbm_open'

The detection in Python 3.10 is aimed only at the existence of the library 
file and does not try to link code: https://github.com/python/cpython/
blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/setup.py#L1430C34-L1430C65

> So I'm really confused. How does it work in Python 3.10 ? Did we already
> need the historic dbm interface back then ?

Me, too, but only because my knowledge about C functions, macro expansion 
etc. is very limited, I can not comment your analysis why the dbm module 
works despite missing symbols. Anyway, with Python 3.11 the dbm module is 
not build at all atm.

Regards, Bernd

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/berkeleydb: add option for historic dbm interface

[Thomas Petazzoni via buildroot]

Hello,

On Tue, 08 Aug 2023 18:01:40 +0200
Bernd Kuhls <bernd@kuhls.net> wrote:

> the problem I am fixing is only about the detection during configure and 
> unrelated to the code of the dbm module.

I understand, but it raises additional questions.

> The detection logic in Python 3.11 tries to link a sample program 
> containing "dbm_open()" to -ldb: https://github.com/python/cpython/blob/
> d2340ef25721b6a72d45d4508c672c4be38c67d3/configure.ac#L4007
> 
> and fails, for whatever reason:
> conftest.c:(.text.startup+0x8): undefined reference to `__db_ndbm_open'

Absolutely, this is understood.

> The detection in Python 3.10 is aimed only at the existence of the library 
> file and does not try to link code: https://github.com/python/cpython/
> blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/setup.py#L1430C34-L1430C65

Correct.

> > So I'm really confused. How does it work in Python 3.10 ? Did we already
> > need the historic dbm interface back then ?  
> 
> Me, too, but only because my knowledge about C functions, macro expansion 
> etc. is very limited, I can not comment your analysis why the dbm module 
> works despite missing symbols. Anyway, with Python 3.11 the dbm module is 
> not build at all atm.

Yes, I understand that, but rather than "blindly" fixing the issue, I'd
like to understand what is happening. Indeed, as I explained in
previous e-mail, the dbm module in Python 3.10 is *already* using
__db_ndbm_open(), which isn't provided by libdb-5.3.so, so I'm really
confused on how it can work today. Perhaps we have an existing issue in
Python 3.10, and your change needs to also be backported to our
maintenance branch? This is why understand what's happening is
important.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/berkeleydb: add option for historic dbm interface

[Peter Korsgaard]

>>>>> "Bernd" == Bernd Kuhls <bernd@kuhls.net> writes:

 > Berkeley DB support in python depends on dbm, when missing python fails
 > to detect libdb:
 > https://github.com/python/cpython/blob/d2340ef25721b6a72d45d4508c672c4be38c67d3/configure.ac#L4002

 > quoting python configure log:
 > checking for libdb... no

 > quoting python config.log:
 > conftest.c:(.text.startup+0x8): undefined reference to `__db_ndbm_open'

 > This patch provides a configure option for the historic dbm interface
 > to be used by the python package.

 > Signed-off-by: Bernd Kuhls <bernd@kuhls.net>

Committed to 2023.02.x and 2023.05.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/python3: berkeleydb support needs the dbm interface

[Peter Korsgaard]

>>>>> "Bernd" == Bernd Kuhls <bernd@kuhls.net> writes:

 > Berkeley DB support in python depends on dbm, when missing python fails
 > to detect libdb:
 > https://github.com/python/cpython/blob/d2340ef25721b6a72d45d4508c672c4be38c67d3/configure.ac#L4002

 > quoting configure log:
 > checking for libdb... no

 > quoting python config.log:
 > conftest.c:(.text.startup+0x8): undefined reference to `__db_ndbm_open'

 > Only with this patch python builds a berkeleydb module:

 >   output/build/python3-3.11.4$ find -iname *dbm*.so
 >   ./build/lib.linux-x86_64-3.11/_dbm.cpython-311-x86_64-linux-gnu.so

 > since python 3.11 (added with buildroot commit
 > 738500c296c8b1206f20e94ca3e7c5932a6a0486) changed the way it detects libdb:
 > https://github.com/python/cpython/commit/0a9f69539be27acf1cddf1b58d02a88d02e5008d

 > Signed-off-by: Bernd Kuhls <bernd@kuhls.net>

Committed to 2023.02.x and 2023.05.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot