* [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket
@ 2019-11-11 22:17 gavinli
2019-11-12 7:28 ` Mikko.Rapeli
0 siblings, 1 reply; 6+ messages in thread
From: gavinli @ 2019-11-11 22:17 UTC (permalink / raw)
To: bitbake-devel; +Cc: Gavin Li
From: Gavin Li <gavin@matician.com>
With PRSERV_HOST = "localhost:0", this message would occasionally pop up
during the initial cache read:
WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
value = pickled.load()
The file location stated is irrelevant; it just happens to be wherever
CPython decides to run the garbage collector. The issue is that after we
fork off a PRServer, self.socket is also duplicated. The parent side of
it also needs to be closed.
Signed-off-by: Gavin Li <gavin@matician.com>
---
bitbake/lib/prserv/serv.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index be3acec36a..1d9148b817 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
try:
pid = os.fork()
if pid > 0:
+ self.socket.close() # avoid ResourceWarning in parent
return pid
except OSError as e:
raise Exception("%s [%d]" % (e.strerror, e.errno))
--
2.23.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket
2019-11-11 22:17 [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket gavinli
@ 2019-11-12 7:28 ` Mikko.Rapeli
2019-11-12 22:19 ` Peter Kjellerstedt
0 siblings, 1 reply; 6+ messages in thread
From: Mikko.Rapeli @ 2019-11-12 7:28 UTC (permalink / raw)
To: gavinli; +Cc: gavin, bitbake-devel
Hi,
On Mon, Nov 11, 2019 at 02:17:39PM -0800, gavinli@thegavinli.com wrote:
> From: Gavin Li <gavin@matician.com>
>
> With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> during the initial cache read:
Thanks for this fix!
It's not occasionally, it's all the time with zeus and Debian build container...
If it helps:
Tested-by: Mikko Rapeli <mikko.rapeli@bmw.de>
-Mikko
> WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
> value = pickled.load()
>
> The file location stated is irrelevant; it just happens to be wherever
> CPython decides to run the garbage collector. The issue is that after we
> fork off a PRServer, self.socket is also duplicated. The parent side of
> it also needs to be closed.
>
> Signed-off-by: Gavin Li <gavin@matician.com>
> ---
> bitbake/lib/prserv/serv.py | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index be3acec36a..1d9148b817 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
> try:
> pid = os.fork()
> if pid > 0:
> + self.socket.close() # avoid ResourceWarning in parent
> return pid
> except OSError as e:
> raise Exception("%s [%d]" % (e.strerror, e.errno))
> --
> 2.23.0
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket
2019-11-12 7:28 ` Mikko.Rapeli
@ 2019-11-12 22:19 ` Peter Kjellerstedt
0 siblings, 0 replies; 6+ messages in thread
From: Peter Kjellerstedt @ 2019-11-12 22:19 UTC (permalink / raw)
To: Mikko.Rapeli, gavinli; +Cc: gavin, bitbake-devel
> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
> bounces@lists.openembedded.org> On Behalf Of Mikko.Rapeli@bmw.de
> Sent: den 12 november 2019 08:29
> To: gavinli@thegavinli.com
> Cc: gavin@matician.com; bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH] bitbake: prserv: fix ResourceWarning
> due to unclosed socket
>
> Hi,
>
> On Mon, Nov 11, 2019 at 02:17:39PM -0800, gavinli@thegavinli.com wrote:
> > From: Gavin Li <gavin@matician.com>
> >
> > With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> > during the initial cache read:
>
> Thanks for this fix!
Yes, finally! I've been hunting this for quite a while...
> It's not occasionally, it's all the time with zeus and Debian build
> container...
It is occasional for Warrior, so I guess it should be backported to
both Zeus and Warrior.
> If it helps:
> Tested-by: Mikko Rapeli <mikko.rapeli@bmw.de>
>
> -Mikko
//Peter
> > WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446:
> ResourceWarning: unclosed <socket.socket fd=10,
> family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 45655)>
> > value = pickled.load()
> >
> > The file location stated is irrelevant; it just happens to be wherever
> > CPython decides to run the garbage collector. The issue is that after we
> > fork off a PRServer, self.socket is also duplicated. The parent side of
> > it also needs to be closed.
> >
> > Signed-off-by: Gavin Li <gavin@matician.com>
> > ---
> > bitbake/lib/prserv/serv.py | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> > index be3acec36a..1d9148b817 100644
> > --- a/bitbake/lib/prserv/serv.py
> > +++ b/bitbake/lib/prserv/serv.py
> > @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
> > try:
> > pid = os.fork()
> > if pid > 0:
> > + self.socket.close() # avoid ResourceWarning in parent
> > return pid
> > except OSError as e:
> > raise Exception("%s [%d]" % (e.strerror, e.errno))
> > --
> > 2.23.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket
@ 2019-11-11 22:15 gavinli
2019-11-20 16:11 ` Martin Jansa
0 siblings, 1 reply; 6+ messages in thread
From: gavinli @ 2019-11-11 22:15 UTC (permalink / raw)
To: openembedded-core; +Cc: Gavin Li
From: Gavin Li <gavin@matician.com>
With PRSERV_HOST = "localhost:0", this message would occasionally pop up
during the initial cache read:
WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
value = pickled.load()
The file location stated is irrelevant; it just happens to be wherever
CPython decides to run the garbage collector. The issue is that after we
fork off a PRServer, self.socket is also duplicated. The parent side of
it also needs to be closed.
Signed-off-by: Gavin Li <gavin@matician.com>
---
bitbake/lib/prserv/serv.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index be3acec36a..1d9148b817 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
try:
pid = os.fork()
if pid > 0:
+ self.socket.close() # avoid ResourceWarning in parent
return pid
except OSError as e:
raise Exception("%s [%d]" % (e.strerror, e.errno))
--
2.23.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket
2019-11-11 22:15 gavinli
@ 2019-11-20 16:11 ` Martin Jansa
2019-11-23 14:01 ` Richard Purdie
0 siblings, 1 reply; 6+ messages in thread
From: Martin Jansa @ 2019-11-20 16:11 UTC (permalink / raw)
To: gavinli; +Cc: Gavin Li, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 2313 bytes --]
Thanks, but it still starts and terminates PRServer immediately, right?
With your fix, all my builds now start with:
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 34335, PID: 9429
NOTE: Terminating PRServer...
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 45753, PID: 9433
before this I was using:
https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
which doesn't start it just to terminate it, but RP said that my fix
doesn't look correct, so I haven't sent it to ML. BTW: bitbake patches
should go to bitbake-devel ML not oe-core ML.
On Mon, Nov 11, 2019 at 11:15 PM <gavinli@thegavinli.com> wrote:
> From: Gavin Li <gavin@matician.com>
>
> With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> during the initial cache read:
>
> WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446:
> ResourceWarning: unclosed <socket.socket fd=10,
> family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 45655)>
> value = pickled.load()
>
> The file location stated is irrelevant; it just happens to be wherever
> CPython decides to run the garbage collector. The issue is that after we
> fork off a PRServer, self.socket is also duplicated. The parent side of
> it also needs to be closed.
>
> Signed-off-by: Gavin Li <gavin@matician.com>
> ---
> bitbake/lib/prserv/serv.py | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index be3acec36a..1d9148b817 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
> try:
> pid = os.fork()
> if pid > 0:
> + self.socket.close() # avoid ResourceWarning in parent
> return pid
> except OSError as e:
> raise Exception("%s [%d]" % (e.strerror, e.errno))
> --
> 2.23.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
[-- Attachment #2: Type: text/html, Size: 3288 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket
2019-11-20 16:11 ` Martin Jansa
@ 2019-11-23 14:01 ` Richard Purdie
0 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2019-11-23 14:01 UTC (permalink / raw)
To: Martin Jansa, gavinli
Cc: Gavin Li, Patches and discussions about the oe-core layer
On Wed, 2019-11-20 at 17:11 +0100, Martin Jansa wrote:
> Thanks, but it still starts and terminates PRServer immediately,
> right?
>
> With your fix, all my builds now start with:
>
> NOTE: Started PRServer with DBfile: /OE/build/oe-
> core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 34335, PID: 9429
> NOTE: Terminating PRServer...
> NOTE: Started PRServer with DBfile: /OE/build/oe-
> core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 45753, PID: 9433
>
> before this I was using:
> https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
> which doesn't start it just to terminate it, but RP said that my fix
> doesn't look correct, so I haven't sent it to ML. BTW: bitbake
> patches should go to bitbake-devel ML not oe-core ML.
FWIW I finally got around to fixing this. The resource issue was half
the problem, the unnecessary restarts are now also addressed. This was
particularly important to get right for the context of memory resident
bitbake.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-11-23 14:01 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-11 22:17 [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket gavinli
2019-11-12 7:28 ` Mikko.Rapeli
2019-11-12 22:19 ` Peter Kjellerstedt
-- strict thread matches above, loose matches on Subject: below --
2019-11-11 22:15 gavinli
2019-11-20 16:11 ` Martin Jansa
2019-11-23 14:01 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.