[PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[gavinli]

From: Gavin Li <gavin@matician.com>

With PRSERV_HOST = "localhost:0", this message would occasionally pop up
during the initial cache read:

WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
  value = pickled.load()

The file location stated is irrelevant; it just happens to be wherever
CPython decides to run the garbage collector. The issue is that after we
fork off a PRServer, self.socket is also duplicated. The parent side of
it also needs to be closed.

Signed-off-by: Gavin Li <gavin@matician.com>
---
 bitbake/lib/prserv/serv.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index be3acec36a..1d9148b817 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
         try:
             pid = os.fork()
             if pid > 0:
+                self.socket.close() # avoid ResourceWarning in parent
                 return pid
         except OSError as e:
             raise Exception("%s [%d]" % (e.strerror, e.errno))
-- 
2.23.0

Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[Mikko.Rapeli]

Hi,

On Mon, Nov 11, 2019 at 02:17:39PM -0800, gavinli@thegavinli.com wrote:
> From: Gavin Li <gavin@matician.com>
> 
> With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> during the initial cache read:

Thanks for this fix!

It's not occasionally, it's all the time with zeus and Debian build container...

If it helps:
Tested-by: Mikko Rapeli <mikko.rapeli@bmw.de>

-Mikko

> WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
>   value = pickled.load()
> 
> The file location stated is irrelevant; it just happens to be wherever
> CPython decides to run the garbage collector. The issue is that after we
> fork off a PRServer, self.socket is also duplicated. The parent side of
> it also needs to be closed.
> 
> Signed-off-by: Gavin Li <gavin@matician.com>
> ---
>  bitbake/lib/prserv/serv.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index be3acec36a..1d9148b817 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
>          try:
>              pid = os.fork()
>              if pid > 0:
> +                self.socket.close() # avoid ResourceWarning in parent
>                  return pid
>          except OSError as e:
>              raise Exception("%s [%d]" % (e.strerror, e.errno))
> -- 
> 2.23.0
> 
> -- 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
> bounces@lists.openembedded.org> On Behalf Of Mikko.Rapeli@bmw.de
> Sent: den 12 november 2019 08:29
> To: gavinli@thegavinli.com
> Cc: gavin@matician.com; bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH] bitbake: prserv: fix ResourceWarning
> due to unclosed socket
> 
> Hi,
> 
> On Mon, Nov 11, 2019 at 02:17:39PM -0800, gavinli@thegavinli.com wrote:
> > From: Gavin Li <gavin@matician.com>
> >
> > With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> > during the initial cache read:
> 
> Thanks for this fix!

Yes, finally! I've been hunting this for quite a while...

> It's not occasionally, it's all the time with zeus and Debian build
> container...

It is occasional for Warrior, so I guess it should be backported to 
both Zeus and Warrior.

> If it helps:
> Tested-by: Mikko Rapeli <mikko.rapeli@bmw.de>
> 
> -Mikko

//Peter

> > WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446:
> ResourceWarning: unclosed <socket.socket fd=10,
> family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 45655)>
> >   value = pickled.load()
> >
> > The file location stated is irrelevant; it just happens to be wherever
> > CPython decides to run the garbage collector. The issue is that after we
> > fork off a PRServer, self.socket is also duplicated. The parent side of
> > it also needs to be closed.
> >
> > Signed-off-by: Gavin Li <gavin@matician.com>
> > ---
> >  bitbake/lib/prserv/serv.py | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> > index be3acec36a..1d9148b817 100644
> > --- a/bitbake/lib/prserv/serv.py
> > +++ b/bitbake/lib/prserv/serv.py
> > @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
> >          try:
> >              pid = os.fork()
> >              if pid > 0:
> > +                self.socket.close() # avoid ResourceWarning in parent
> >                  return pid
> >          except OSError as e:
> >              raise Exception("%s [%d]" % (e.strerror, e.errno))
> > --
> > 2.23.0

Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[Richard Purdie]

On Wed, 2019-11-20 at 17:11 +0100, Martin Jansa wrote:
> Thanks, but it still starts and terminates PRServer immediately,
> right?
> 
> With your fix, all my builds now start with:
> 
> NOTE: Started PRServer with DBfile: /OE/build/oe-
> core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 34335, PID: 9429
> NOTE: Terminating PRServer...
> NOTE: Started PRServer with DBfile: /OE/build/oe-
> core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 45753, PID: 9433
> 
> before this I was using:
> https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
> which doesn't start it just to terminate it, but RP said that my fix
> doesn't look correct, so I haven't sent it to ML. BTW: bitbake
> patches should go to bitbake-devel ML not oe-core ML.

FWIW I finally got around to fixing this. The resource issue was half
the problem, the unnecessary restarts are now also addressed. This was
particularly important to get right for the context of memory resident
bitbake.

Cheers,

Richard

Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2313 bytes --]

Thanks, but it still starts and terminates PRServer immediately, right?

With your fix, all my builds now start with:

NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 34335, PID: 9429
NOTE: Terminating PRServer...
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 45753, PID: 9433

before this I was using:
https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
which doesn't start it just to terminate it, but RP said that my fix
doesn't look correct, so I haven't sent it to ML. BTW: bitbake patches
should go to bitbake-devel ML not oe-core ML.


On Mon, Nov 11, 2019 at 11:15 PM <gavinli@thegavinli.com> wrote:

> From: Gavin Li <gavin@matician.com>
>
> With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> during the initial cache read:
>
> WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446:
> ResourceWarning: unclosed <socket.socket fd=10,
> family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 45655)>
>   value = pickled.load()
>
> The file location stated is irrelevant; it just happens to be wherever
> CPython decides to run the garbage collector. The issue is that after we
> fork off a PRServer, self.socket is also duplicated. The parent side of
> it also needs to be closed.
>
> Signed-off-by: Gavin Li <gavin@matician.com>
> ---
>  bitbake/lib/prserv/serv.py | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index be3acec36a..1d9148b817 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
>          try:
>              pid = os.fork()
>              if pid > 0:
> +                self.socket.close() # avoid ResourceWarning in parent
>                  return pid
>          except OSError as e:
>              raise Exception("%s [%d]" % (e.strerror, e.errno))
> --
> 2.23.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3288 bytes --]

[PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[gavinli]

From: Gavin Li <gavin@matician.com>

With PRSERV_HOST = "localhost:0", this message would occasionally pop up
during the initial cache read:

WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
  value = pickled.load()

The file location stated is irrelevant; it just happens to be wherever
CPython decides to run the garbage collector. The issue is that after we
fork off a PRServer, self.socket is also duplicated. The parent side of
it also needs to be closed.

Signed-off-by: Gavin Li <gavin@matician.com>
---
 bitbake/lib/prserv/serv.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index be3acec36a..1d9148b817 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
         try:
             pid = os.fork()
             if pid > 0:
+                self.socket.close() # avoid ResourceWarning in parent
                 return pid
         except OSError as e:
             raise Exception("%s [%d]" % (e.strerror, e.errno))
-- 
2.23.0