[RFC][PATCH] distutils: put tests to a separate package

[RFC][PATCH] distutils: put tests to a separate package

[Konrad Weihmann]

Put files from installed tests into a separate package to avoid having
them in an image unwanted and to avoid having more runtime
dependencies than necessary.

Background:
As setuptools is pretty much unregulated in terms what is installed
by some module, even tests are installed into base package, which then
go right into production, although completely unnecessary.
When now trying to determine the needed runtime dependencies, more
than actually needed will slip into the image.
To give a really bad example GitPython ships its tests into production
leading to the effect that theoretically besides the known RDEPENDS
python3-asyncio, python3-distutils, python3-idle, python3-misc,
python3-pickle, python3-shell, python3-ddt and python3-nose would be
required.
That's why I came up with the idea to package tests (if existing) into
a separate package, which is NOT installed by default.
Please let me know if you see any issues with that patch

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
---
 meta/classes/distutils-common-base.bbclass | 5 +++++
 meta/classes/distutils3-base.bbclass       | 1 +
 2 files changed, 6 insertions(+)

diff --git a/meta/classes/distutils-common-base.bbclass b/meta/classes/distutils-common-base.bbclass
index 94b5fd426d..50dca7550b 100644
--- a/meta/classes/distutils-common-base.bbclass
+++ b/meta/classes/distutils-common-base.bbclass
@@ -13,8 +13,13 @@ export CCSHARED  = "-fPIC -DPIC"
 # the python executable
 export LINKFORSHARED = "{SECURITY_CFLAGS} -Xlinker -export-dynamic"
 
+PACKAGES =+ "${@oe.utils.ifelse(bb.data.inherits_class('image', d), '', '${PN}-tests')}"
+
 FILES_${PN} += "${libdir}/* ${libdir}/${PYTHON_DIR}/*"
 
+FILES_${PN}-tests += "${PYTHON_SITEPACKAGES_DIR}/**/test/*"
+ALLOW_EMPTY_${PN}-tests = "1"
+
 FILES_${PN}-staticdev += "\
   ${PYTHON_SITEPACKAGES_DIR}/*.a \
 "
diff --git a/meta/classes/distutils3-base.bbclass b/meta/classes/distutils3-base.bbclass
index 7dbf07ac4b..54bb963a57 100644
--- a/meta/classes/distutils3-base.bbclass
+++ b/meta/classes/distutils3-base.bbclass
@@ -1,5 +1,6 @@
 DEPENDS  += "${@["${PYTHON_PN}-native ${PYTHON_PN}", ""][(d.getVar('PACKAGES') == '')]}"
 RDEPENDS_${PN} += "${@['', '${PYTHON_PN}-core']['${CLASSOVERRIDE}' == 'class-target']}"
+RDEPENDS_${PN}-tests += "${PN}"
 
 inherit distutils-common-base python3native
 
-- 
2.20.1

Re: [OE-core] [RFC][PATCH] distutils: put tests to a separate package

[Richard Purdie]

On Tue, 2020-07-07 at 21:01 +0200, Konrad Weihmann wrote:
> Put files from installed tests into a separate package to avoid
> having
> them in an image unwanted and to avoid having more runtime
> dependencies than necessary.
> 
> Background:
> As setuptools is pretty much unregulated in terms what is installed
> by some module, even tests are installed into base package, which
> then
> go right into production, although completely unnecessary.
> When now trying to determine the needed runtime dependencies, more
> than actually needed will slip into the image.
> To give a really bad example GitPython ships its tests into
> production
> leading to the effect that theoretically besides the known RDEPENDS
> python3-asyncio, python3-distutils, python3-idle, python3-misc,
> python3-pickle, python3-shell, python3-ddt and python3-nose would be
> required.
> That's why I came up with the idea to package tests (if existing)
> into
> a separate package, which is NOT installed by default.
> Please let me know if you see any issues with that patch

Shouldn't these be part of a PN-ptests package to match the rest of the
system?

Cheers,

Richard

Re: [OE-core] [RFC][PATCH] distutils: put tests to a separate package

[Konrad Weihmann]

That's a very good question.
I myself haven't seen python unit tests used via ptests in the wild, but 
if that is used, yes fully agree, it should be rather PN-ptests than 
PN-tests.
BTW it would also require to create a run-ptest file, which would 
complicate things even more.

Regards
Konrad

On 07.07.20 21:25, Richard Purdie wrote:
> On Tue, 2020-07-07 at 21:01 +0200, Konrad Weihmann wrote:
>> Put files from installed tests into a separate package to avoid
>> having
>> them in an image unwanted and to avoid having more runtime
>> dependencies than necessary.
>>
>> Background:
>> As setuptools is pretty much unregulated in terms what is installed
>> by some module, even tests are installed into base package, which
>> then
>> go right into production, although completely unnecessary.
>> When now trying to determine the needed runtime dependencies, more
>> than actually needed will slip into the image.
>> To give a really bad example GitPython ships its tests into
>> production
>> leading to the effect that theoretically besides the known RDEPENDS
>> python3-asyncio, python3-distutils, python3-idle, python3-misc,
>> python3-pickle, python3-shell, python3-ddt and python3-nose would be
>> required.
>> That's why I came up with the idea to package tests (if existing)
>> into
>> a separate package, which is NOT installed by default.
>> Please let me know if you see any issues with that patch
> 
> Shouldn't these be part of a PN-ptests package to match the rest of the
> system?
> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: [OE-core] [RFC][PATCH] distutils: put tests to a separate package

[Richard Purdie]

On Tue, 2020-07-07 at 21:34 +0200, Konrad Weihmann wrote:
> That's a very good question.
> I myself haven't seen python unit tests used via ptests in the wild,
> but 
> if that is used, yes fully agree, it should be rather PN-ptests than 
> PN-tests.
> BTW it would also require to create a run-ptest file, which would 
> complicate things even more.

Even if we don't create the run-ptest yet, I think putting them there
is the more correct location...

Cheers,

Richard