[PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[gavinli]

From: Gavin Li <gavin@matician.com>

With PRSERV_HOST = "localhost:0", this message would occasionally pop up
during the initial cache read:

WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
  value = pickled.load()

The file location stated is irrelevant; it just happens to be wherever
CPython decides to run the garbage collector. The issue is that after we
fork off a PRServer, self.socket is also duplicated. The parent side of
it also needs to be closed.

Signed-off-by: Gavin Li <gavin@matician.com>
---
 bitbake/lib/prserv/serv.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index be3acec36a..1d9148b817 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
         try:
             pid = os.fork()
             if pid > 0:
+                self.socket.close() # avoid ResourceWarning in parent
                 return pid
         except OSError as e:
             raise Exception("%s [%d]" % (e.strerror, e.errno))
-- 
2.23.0

✗ patchtest: failure for bitbake: prserv: fix ResourceWarning due to unclosed socket

[Patchwork]

== Series Details ==

Series: bitbake: prserv: fix ResourceWarning due to unclosed socket
Revision: 1
URL   : https://patchwork.openembedded.org/series/21085/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series sent to the wrong mailing list or some patches from the series correspond to different mailing lists [test_target_mailing_list] 
  Suggested fix    Send the series again to the correct mailing list (ML)
  Suggested ML     bitbake-devel@lists.openembedded.org [http://git.openembedded.org/bitbake/]
  Patch's path:    bitbake/lib/prserv/serv.py

* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 4dc804b2de)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2313 bytes --]

Thanks, but it still starts and terminates PRServer immediately, right?

With your fix, all my builds now start with:

NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 34335, PID: 9429
NOTE: Terminating PRServer...
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 45753, PID: 9433

before this I was using:
https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
which doesn't start it just to terminate it, but RP said that my fix
doesn't look correct, so I haven't sent it to ML. BTW: bitbake patches
should go to bitbake-devel ML not oe-core ML.


On Mon, Nov 11, 2019 at 11:15 PM <gavinli@thegavinli.com> wrote:

> From: Gavin Li <gavin@matician.com>
>
> With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> during the initial cache read:
>
> WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446:
> ResourceWarning: unclosed <socket.socket fd=10,
> family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 45655)>
>   value = pickled.load()
>
> The file location stated is irrelevant; it just happens to be wherever
> CPython decides to run the garbage collector. The issue is that after we
> fork off a PRServer, self.socket is also duplicated. The parent side of
> it also needs to be closed.
>
> Signed-off-by: Gavin Li <gavin@matician.com>
> ---
>  bitbake/lib/prserv/serv.py | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index be3acec36a..1d9148b817 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
>          try:
>              pid = os.fork()
>              if pid > 0:
> +                self.socket.close() # avoid ResourceWarning in parent
>                  return pid
>          except OSError as e:
>              raise Exception("%s [%d]" % (e.strerror, e.errno))
> --
> 2.23.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3288 bytes --]

Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[Richard Purdie]

On Wed, 2019-11-20 at 17:11 +0100, Martin Jansa wrote:
> Thanks, but it still starts and terminates PRServer immediately,
> right?
> 
> With your fix, all my builds now start with:
> 
> NOTE: Started PRServer with DBfile: /OE/build/oe-
> core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 34335, PID: 9429
> NOTE: Terminating PRServer...
> NOTE: Started PRServer with DBfile: /OE/build/oe-
> core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 45753, PID: 9433
> 
> before this I was using:
> https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
> which doesn't start it just to terminate it, but RP said that my fix
> doesn't look correct, so I haven't sent it to ML. BTW: bitbake
> patches should go to bitbake-devel ML not oe-core ML.

FWIW I finally got around to fixing this. The resource issue was half
the problem, the unnecessary restarts are now also addressed. This was
particularly important to get right for the context of memory resident
bitbake.

Cheers,

Richard