[PATCH 1/2] utils: Add disable_network function

[PATCH 1/2] utils: Add disable_network function

[Richard Purdie]

Add a function which uses the unshare glibc call to disable networking
in the current process. This doesn't work on older distros/kernels
but will on more recent ones so for now we simply ignore the cases we
can't execute on. uid/gid can be passed in externally so this can
work with pseudo/fakeroot contexts.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/utils.py | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 1a51589704..0312231933 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -27,6 +27,7 @@ import errno
 import signal
 import collections
 import copy
+import ctypes
 from subprocess import getstatusoutput
 from contextlib import contextmanager
 from ctypes import cdll
@@ -1595,6 +1596,36 @@ def set_process_name(name):
     except:
         pass
 
+def disable_network(uid=None, gid=None):
+    """
+    Disable networking in the current process if the kernel supports it, else
+    just return after logging to debug. To do this we need to create a new user
+    namespace, then map back to the original uid/gid.
+    """
+    libc = ctypes.CDLL('libc.so.6')
+
+    # From sched.h
+    # New user namespace
+    CLONE_NEWUSER = 0x10000000
+    # New network namespace
+    CLONE_NEWNET = 0x40000000
+
+    if uid is None:
+        uid = os.getuid()
+    if gid is None:
+        gid = os.getgid()
+
+    ret = libc.unshare(CLONE_NEWNET | CLONE_NEWUSER)
+    if ret != 0:
+        logger.debug("System doesn't suport disabling network without admin privs")
+        return
+    with open("/proc/self/uid_map", "w") as f:
+        f.write("%s %s 1" % (uid, uid))
+    with open("/proc/self/setgroups", "w") as f:
+        f.write("deny")
+    with open("/proc/self/gid_map", "w") as f:
+        f.write("%s %s 1" % (gid, gid))
+
 def export_proxies(d):
     """ export common proxies variables from datastore to environment """
     import os
-- 
2.32.0

[PATCH 2/2] bitbake-worker: Add/support network task flag

[Richard Purdie]

This patch changes behaviour. After this change any task which does not
have the network flag set will have networking disabled on systems that
support that (kernel version dependent).

Add a "network" task specific flag which then triggers networking to
be enabled for this task, it is otherwise disabled.

This needs to happen before we enter the fakeroot environment of the task
due to the need for the real uid/gid which we save in the parent process.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 bin/bitbake-worker | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/bin/bitbake-worker b/bin/bitbake-worker
index bf96207edc..3aaf3c2444 100755
--- a/bin/bitbake-worker
+++ b/bin/bitbake-worker
@@ -152,6 +152,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskha
     fakeenv = {}
     umask = None
 
+    uid = os.getuid()
+    gid = os.getgid()
+
+
     taskdep = workerdata["taskdeps"][fn]
     if 'umask' in taskdep and taskname in taskdep['umask']:
         umask = taskdep['umask'][taskname]
@@ -257,6 +261,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskha
 
                 bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN"), taskname.replace("do_", "")))
 
+                if not the_data.getVarFlag(taskname, 'network', False):
+                    logger.debug("Attempting to disable network")
+                    bb.utils.disable_network(uid, gid)
+
                 # exported_vars() returns a generator which *cannot* be passed to os.environ.update() 
                 # successfully. We also need to unset anything from the environment which shouldn't be there 
                 exports = bb.data.exported_vars(the_data)
-- 
2.32.0

Re: [bitbake-devel] [PATCH 2/2] bitbake-worker: Add/support network task flag

[Robert Yang]

Hi RP,

On 1/8/22 7:15 AM, Richard Purdie wrote:
> This patch changes behaviour. After this change any task which does not
> have the network flag set will have networking disabled on systems that
> support that (kernel version dependent).
> 
> Add a "network" task specific flag which then triggers networking to
> be enabled for this task, it is otherwise disabled.
> 
> This needs to happen before we enter the fakeroot environment of the task
> due to the need for the real uid/gid which we save in the parent process.
>

It seems that this patch doesn't work with NIS user, since NIS user alwasy try
to access the network to translate UID to USER, but there is no network, so we
will get the errors like the following during the build:

do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable

These error messages may cause build failures such as

1) bitbake quilt-native -cdevshell doesn't work when set OE_TERMINAL = "screen" 
in conf/local.conf:
ERROR: quilt-native-0.66-r0 do_devshell: Unable to spawn terminal screen: 
Execution of /path/to/0.66-r0/temp/run.do_terminal.189055' failed with exit code 1:
getpwuid() can't identify your account!

If no OE_TERMINAL is setting, it will go into a shell with "I have no name!":

I have no name!@host:/path/to/quilt-0.66$

2) bitbake cmake-native failed
This because the error string "do_ypcall: clnt_call: RPC: Unable to send; errno 
= Network is unreachable" has been assigned to a variable, thus caused while 
errors, for example:
VAR=$(whomai)

It will get the string "do_ypcall: clnt_call: RPC: Unable to send; errno = 
Network is unreachable", and cause build errors.

// Robert

> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>   bin/bitbake-worker | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/bin/bitbake-worker b/bin/bitbake-worker
> index bf96207edc..3aaf3c2444 100755
> --- a/bin/bitbake-worker
> +++ b/bin/bitbake-worker
> @@ -152,6 +152,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskha
>       fakeenv = {}
>       umask = None
>   
> +    uid = os.getuid()
> +    gid = os.getgid()
> +
> +
>       taskdep = workerdata["taskdeps"][fn]
>       if 'umask' in taskdep and taskname in taskdep['umask']:
>           umask = taskdep['umask'][taskname]
> @@ -257,6 +261,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskha
>   
>                   bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN"), taskname.replace("do_", "")))
>   
> +                if not the_data.getVarFlag(taskname, 'network', False):
> +                    logger.debug("Attempting to disable network")
> +                    bb.utils.disable_network(uid, gid)
> +
>                   # exported_vars() returns a generator which *cannot* be passed to os.environ.update()
>                   # successfully. We also need to unset anything from the environment which shouldn't be there
>                   exports = bb.data.exported_vars(the_data)
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13225): https://lists.openembedded.org/g/bitbake-devel/message/13225
> Mute This Topic: https://lists.openembedded.org/mt/88273738/3616940
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [liezhi.yang@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [bitbake-devel] [PATCH 2/2] bitbake-worker: Add/support network task flag

[Richard Purdie]

On Wed, 2022-01-19 at 19:49 +0800, Robert Yang wrote:
> Hi RP,
> 
> On 1/8/22 7:15 AM, Richard Purdie wrote:
> > This patch changes behaviour. After this change any task which does not
> > have the network flag set will have networking disabled on systems that
> > support that (kernel version dependent).
> > 
> > Add a "network" task specific flag which then triggers networking to
> > be enabled for this task, it is otherwise disabled.
> > 
> > This needs to happen before we enter the fakeroot environment of the task
> > due to the need for the real uid/gid which we save in the parent process.
> > 
> 
> It seems that this patch doesn't work with NIS user, since NIS user alwasy try
> to access the network to translate UID to USER, but there is no network, so we
> will get the errors like the following during the build:
> 
> do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
> 
> These error messages may cause build failures such as
> 
> 1) bitbake quilt-native -cdevshell doesn't work when set OE_TERMINAL = "screen" 
> in conf/local.conf:
> ERROR: quilt-native-0.66-r0 do_devshell: Unable to spawn terminal screen: 
> Execution of /path/to/0.66-r0/temp/run.do_terminal.189055' failed with exit code 1:
> getpwuid() can't identify your account!
> 
> If no OE_TERMINAL is setting, it will go into a shell with "I have no name!":
> 
> I have no name!@host:/path/to/quilt-0.66$
> 
> 2) bitbake cmake-native failed
> This because the error string "do_ypcall: clnt_call: RPC: Unable to send; errno 
> = Network is unreachable" has been assigned to a variable, thus caused while 
> errors, for example:
> VAR=$(whomai)
> 
> It will get the string "do_ypcall: clnt_call: RPC: Unable to send; errno = 
> Network is unreachable", and cause build errors.


Thanks for the report, that is rather sad/unfortunate.

I guess we'll have to add some way to allow network access globally for such
machines, much as I'd prefer not to.

Cheers,

Richard

Re: [bitbake-devel] [PATCH 2/2] bitbake-worker: Add/support network task flag

[Jose Quaresma]

[-- Attachment #1: Type: text/plain, Size: 2993 bytes --]

Hi Richard,

Unfortunately this patch breaks the usage of the icecc.bbclass.

Currently I build with icecc inside a container with network isolation and
my icecc demon runs outside of the container.
The only thing I need to do for using the icecc inside my build container is
mounting the unix /var/run/icecc/iceccd.socket inside the container.

I think that we need something like this functionality to have access to
some sockets connections inside the tasks with the new namespace.

Jose


Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia sexta,
7/01/2022 à(s) 23:15:

> This patch changes behaviour. After this change any task which does not
> have the network flag set will have networking disabled on systems that
> support that (kernel version dependent).
>
> Add a "network" task specific flag which then triggers networking to
> be enabled for this task, it is otherwise disabled.
>
> This needs to happen before we enter the fakeroot environment of the task
> due to the need for the real uid/gid which we save in the parent process.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  bin/bitbake-worker | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/bin/bitbake-worker b/bin/bitbake-worker
> index bf96207edc..3aaf3c2444 100755
> --- a/bin/bitbake-worker
> +++ b/bin/bitbake-worker
> @@ -152,6 +152,10 @@ def fork_off_task(cfg, data, databuilder, workerdata,
> fn, task, taskname, taskha
>      fakeenv = {}
>      umask = None
>
> +    uid = os.getuid()
> +    gid = os.getgid()
> +
> +
>      taskdep = workerdata["taskdeps"][fn]
>      if 'umask' in taskdep and taskname in taskdep['umask']:
>          umask = taskdep['umask'][taskname]
> @@ -257,6 +261,10 @@ def fork_off_task(cfg, data, databuilder, workerdata,
> fn, task, taskname, taskha
>
>                  bb.utils.set_process_name("%s:%s" %
> (the_data.getVar("PN"), taskname.replace("do_", "")))
>
> +                if not the_data.getVarFlag(taskname, 'network', False):
> +                    logger.debug("Attempting to disable network")
> +                    bb.utils.disable_network(uid, gid)
> +
>                  # exported_vars() returns a generator which *cannot* be
> passed to os.environ.update()
>                  # successfully. We also need to unset anything from the
> environment which shouldn't be there
>                  exports = bb.data.exported_vars(the_data)
> --
> 2.32.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13225):
> https://lists.openembedded.org/g/bitbake-devel/message/13225
> Mute This Topic: https://lists.openembedded.org/mt/88273738/5052612
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> quaresma.jose@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
Best regards,

José Quaresma

[-- Attachment #2: Type: text/html, Size: 4346 bytes --]

Re: [bitbake-devel] [PATCH 2/2] bitbake-worker: Add/support network task flag

[Jose Quaresma]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quarta,
19/01/2022 à(s) 23:27:

> On Wed, 2022-01-19 at 23:16 +0000, Jose Quaresma wrote:
> > Unfortunately this patch breaks the usage of the icecc.bbclass.
> >
> > Currently I build with icecc inside a container with network isolation
> and
> > my icecc demon runs outside of the container.
> > The only thing I need to do for using the icecc inside my build
> container is
> > mounting the unix /var/run/icecc/iceccd.socket inside the container.
> >
> > I think that we need something like this functionality to have access to
> > some sockets connections inside the tasks with the new namespace.
>
> I'm open to proposals but the unshare functionality is limited so I
> suspect we
> can't just allow/disallow specific sockets unfortunately. We may just need
> the
> global "allow" flag for the reasons Robert mentioned which would also work
> here,
> at the code of not showing up other network accesses.
>
> Cheers,
>
> Richard
>
>
I will do some more investigation on that to see how we can use some
pre-existing unix sockets from the old namespace.
I think that the IPC namespace is used for that purpose but I need to
understand more about this subject.

-- 
Best regards,

José Quaresma

[-- Attachment #2: Type: text/html, Size: 1824 bytes --]

Re: [bitbake-devel] [PATCH 2/2] bitbake-worker: Add/support network task flag

[Robert Yang]

On 1/20/22 7:27 AM, Richard Purdie wrote:
> On Wed, 2022-01-19 at 23:16 +0000, Jose Quaresma wrote:
>> Unfortunately this patch breaks the usage of the icecc.bbclass.
>>
>> Currently I build with icecc inside a container with network isolation and
>> my icecc demon runs outside of the container.
>> The only thing I need to do for using the icecc inside my build container is
>> mounting the unix /var/run/icecc/iceccd.socket inside the container.
>>
>> I think that we need something like this functionality to have access to
>> some sockets connections inside the tasks with the new namespace.
> 
> I'm open to proposals but the unshare functionality is limited so I suspect we
> can't just allow/disallow specific sockets unfortunately. We may just need the
> global "allow" flag for the reasons Robert mentioned which would also work here,
> at the code of not showing up other network accesses.

I will send 2 patches to make we can global allow network accesses, one is for
bitbake, the other one for oe-core.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13249): https://lists.openembedded.org/g/bitbake-devel/message/13249
> Mute This Topic: https://lists.openembedded.org/mt/88273738/3616940
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [liezhi.yang@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>