loosing netdevices with namespaces and unshare?

loosing netdevices with namespaces and unshare?

[Harald Welte]

Hi all,

I know I'm kind of late to the party in terms of deeper exploration of
Linux network namespaces.  Also, I'm not sure if the netdev list is the
riight place to ask, but a moderate amount of web searching didn't bring
up a solution in multiple hours, and it seems like I could trigger the
kernel (4.11.0) to loose netdevices, which I think is a serious issue.

What I'm doing:
* start a process using the 'unshare' command line tool provided with
  util-linux, e.g. "unshare -nUr bash". I do this as a non-privileged
  user but now that is mapped to uid '0' inside the new
  process/namespace, so I can adjust interface configuration.
* I use "echo $$" to get the PID of that bash process.
* On another terminal in a root shell, I use "ip link set eth0 netns $PID"
  in order to move a given physical device into that namespace.
* I then "exit" that bash, which should - to my knowledge - return the
  "eth0" netdev back to the root namespace, as the bash process was the
  only one using that network namespace

But, to the contrary, this doesn't happen.  The unshare-created netns is
gone, but the netdevice did not get moved back to the root namespace
either.  The only hack to get back to the "eth0" device is to unload the
driver and re-load it.

I can reproduce the above without starting any other process inside that
namespace.  I have verified that there are no /proc/*/ns/net symlinks
left pointing to the ID of that namespace.  What am I missing here?  Is
this the intended behavior?

Of course I know I could simply do something like "ip link set eth0
netns 1" from within the namespace before leaving.  But what if the
process is not bash and the process exits abnormally?   I'd consider
that explicit reassignment more like a hack than a proper solution...

Regards,
	Harald

p.s.: In case you're wondering what I'm actually trying to achieve: Find
an easy way to run a single program in an isolated namespace that only
has one physical (usb) ethernet device.  I would like to execute that
program as unprivileged user but still be able to bind to privileged
ports.  And I want to do this using simple command-line tools without
all the bloat and overhead of "container" solutions that have 99% of
features I don't need.  But let that not distract you, I think the
mysteriously disappearing netdevices are a more general and important
issue.

-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: loosing netdevices with namespaces and unshare?

[Cong Wang]

On Tue, May 30, 2017 at 3:07 PM, Harald Welte <laforge@gnumonks.org> wrote:
> But, to the contrary, this doesn't happen.  The unshare-created netns is
> gone, but the netdevice did not get moved back to the root namespace
> either.  The only hack to get back to the "eth0" device is to unload the
> driver and re-load it.


Net namespace simply unregisters all netdevices inside when it is
gone, no matter where they are from. I am pretty sure you can move it
back to root-ns if you want, it is a little tricky because you have to give
the root-ns a name first.


>
> I can reproduce the above without starting any other process inside that
> namespace.  I have verified that there are no /proc/*/ns/net symlinks
> left pointing to the ID of that namespace.  What am I missing here?  Is
> this the intended behavior?

Yes it is.

>
> Of course I know I could simply do something like "ip link set eth0
> netns 1" from within the namespace before leaving.  But what if the
> process is not bash and the process exits abnormally?   I'd consider
> that explicit reassignment more like a hack than a proper solution...

It doesn't make sense to move it back to where it is from, for example,
what if you move a veth0 from netns1 to netns2 and netns1 is gone
before netns2?

Regards.

Re: loosing netdevices with namespaces and unshare?

[Harald Welte]

Hi Cong,

On Tue, May 30, 2017 at 04:18:17PM -0700, Cong Wang wrote:
> On Tue, May 30, 2017 at 3:07 PM, Harald Welte <laforge@gnumonks.org> wrote:
> > But, to the contrary, this doesn't happen.  The unshare-created netns is
> > gone, but the netdevice did not get moved back to the root namespace
> > either.  The only hack to get back to the "eth0" device is to unload the
> > driver and re-load it.
> 
> 
> Net namespace simply unregisters all netdevices inside when it is
> gone, no matter where they are from. 

ah, ok. I missed that part.  Is there a good piece of documentation on
netwokr namespaces that I should read?

> I am pretty sure you can move it back to root-ns if you want, 

Yes, I can explicitly do that, but this of course doesn't work if e.g.
my [single] process in that namespace crashes due to some bug, OOM or
the like.

> it is a little tricky because you have to give the root-ns a name
> first.

It's actually not, as you can just identify the root-ns by pid 1, so
"ip link set $DEV netns 1" will move it back.  As indicated, I'm worried
about the error paths.

> > What am I missing here?  Is this the intended behavior?
> 
> Yes it is.

thanks for your confirmation.  Guess I have to get used to it.

> > Of course I know I could simply do something like "ip link set eth0
> > netns 1" from within the namespace before leaving.  But what if the
> > process is not bash and the process exits abnormally?   I'd consider
> > that explicit reassignment more like a hack than a proper solution...
> 
> It doesn't make sense to move it back to where it is from, for example,
> what if you move a veth0 from netns1 to netns2 and netns1 is gone
> before netns2?

for virtual devices, I would agree.  For physical devices, I think the
default behavior to unregister them is - from my of course very
subjective point of view - quite questionable.

Regards,
	Harald
-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: loosing netdevices with namespaces and unshare?

[Cong Wang]

On Wed, May 31, 2017 at 5:27 AM, Harald Welte <laforge@gnumonks.org> wrote:
> Hi Cong,
>
> On Tue, May 30, 2017 at 04:18:17PM -0700, Cong Wang wrote:
>> On Tue, May 30, 2017 at 3:07 PM, Harald Welte <laforge@gnumonks.org> wrote:
>> > But, to the contrary, this doesn't happen.  The unshare-created netns is
>> > gone, but the netdevice did not get moved back to the root namespace
>> > either.  The only hack to get back to the "eth0" device is to unload the
>> > driver and re-load it.
>>
>>
>> Net namespace simply unregisters all netdevices inside when it is
>> gone, no matter where they are from.
>
> ah, ok. I missed that part.  Is there a good piece of documentation on
> netwokr namespaces that I should read?

I don't know any doc mentioning this.


>> I am pretty sure you can move it back to root-ns if you want,
>
> Yes, I can explicitly do that, but this of course doesn't work if e.g.
> my [single] process in that namespace crashes due to some bug, OOM or
> the like.
>
>> it is a little tricky because you have to give the root-ns a name
>> first.
>
> It's actually not, as you can just identify the root-ns by pid 1, so
> "ip link set $DEV netns 1" will move it back.  As indicated, I'm worried
> about the error paths.
>

Yeah, using PID works too. Unfortunately the whole namespace
is gone too no matter the last process exits normally or not, it is
just refcount'ed.


>> > What am I missing here?  Is this the intended behavior?
>>
>> Yes it is.
>
> thanks for your confirmation.  Guess I have to get used to it.
>
>> > Of course I know I could simply do something like "ip link set eth0
>> > netns 1" from within the namespace before leaving.  But what if the
>> > process is not bash and the process exits abnormally?   I'd consider
>> > that explicit reassignment more like a hack than a proper solution...
>>
>> It doesn't make sense to move it back to where it is from, for example,
>> what if you move a veth0 from netns1 to netns2 and netns1 is gone
>> before netns2?
>
> for virtual devices, I would agree.  For physical devices, I think the
> default behavior to unregister them is - from my of course very
> subjective point of view - quite questionable.

Network namespace does not special-case the physical devices,
it treats them all equally as abstract net devices.

Hope this helps.

Re: loosing netdevices with namespaces and unshare?

[Harald Welte]

Hi Cong,

On Wed, May 31, 2017 at 10:44:53AM -0700, Cong Wang wrote:
> >> Net namespace simply unregisters all netdevices inside when it is
> >> gone, no matter where they are from.
> >
> > ah, ok. I missed that part.  Is there a good piece of documentation on
> > netwokr namespaces that I should read?
> 
> I don't know any doc mentioning this.

That's of course a pity.  I'll see what can be done about amending the
netns related manpage or the like.

> >> > Of course I know I could simply do something like "ip link set eth0
> >> > netns 1" from within the namespace before leaving.  But what if the
> >> > process is not bash and the process exits abnormally?   I'd consider
> >> > that explicit reassignment more like a hack than a proper solution...
> >>
> >> It doesn't make sense to move it back to where it is from, for example,
> >> what if you move a veth0 from netns1 to netns2 and netns1 is gone
> >> before netns2?
> >
> > for virtual devices, I would agree.  For physical devices, I think the
> > default behavior to unregister them is - from my of course very
> > subjective point of view - quite questionable.
> 
> Network namespace does not special-case the physical devices,
> it treats them all equally as abstract net devices.

I hear you, and I understand that of course from a developer point of
view it makes sense to treat all devices the same.  I just wonder if
from an usability point of view this is the best choice.  Virtual
devices can be (re)created at any time, physical not.

I mean, what is the *use case* for loosing any refrence to a physical
network device and unregistering it from the stack?  Is there any API by
which a new netdevice structure can be instantiated on the actual
hardware?  Registering the netdev is what the driver does during
discovering the system hardware.  If there's a method to "automagically"
loose devices, at the very least I wold expect some reasonable method to
resurrect them.  Unloading the kernel module and reloading it is for
sure not elegant, particularly not if you have multiple Ethernet
devices/ports sharing the same driver.

One could e.g. also think of something like a special namespace that
collects all the "orphan" netdevices.  Something analogous to the old
Unix tradition of "pid 1" collecting all the orphan tasks whose parents
died.  Transferring them into that "netdev orphanage" could
automatically set the link down so that no accidential
routing/forwarding of traffic between the devices is possible.

This is just my two cents.  Given my past involvement in Linux
networking I allow myself having an opinion on such matters.  But if the
kernel networking community thinks it is ok to loose all references to a
physical network device due to processes terminating irregularly (which
will happen, as indicated in OOM or software bug cases), then I will of
course have to accept that.

Regards,
	Harald

-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: loosing netdevices with namespaces and unshare?

[Cong Wang]

On Wed, May 31, 2017 at 11:11 AM, Harald Welte <laforge@gnumonks.org> wrote:
> I mean, what is the *use case* for loosing any refrence to a physical
> network device and unregistering it from the stack?  Is there any API by
> which a new netdevice structure can be instantiated on the actual
> hardware?  Registering the netdev is what the driver does during
> discovering the system hardware.  If there's a method to "automagically"
> loose devices, at the very least I wold expect some reasonable method to
> resurrect them.  Unloading the kernel module and reloading it is for
> sure not elegant, particularly not if you have multiple Ethernet
> devices/ports sharing the same driver.
>
> One could e.g. also think of something like a special namespace that
> collects all the "orphan" netdevices.  Something analogous to the old
> Unix tradition of "pid 1" collecting all the orphan tasks whose parents
> died.  Transferring them into that "netdev orphanage" could
> automatically set the link down so that no accidential
> routing/forwarding of traffic between the devices is possible.

Understand.

But you have other choices than using the physical interface
directly in non-root ns, for example, creating a virtual pair and
connect it with the physical one with a bridge. There are various
ways to achieve this.

Re: loosing netdevices with namespaces and unshare?

[Harald Welte]

Hi Cong,

On Wed, May 31, 2017 at 03:40:33PM -0700, Cong Wang wrote:
> But you have other choices than using the physical interface
> directly in non-root ns, for example, creating a virtual pair and
> connect it with the physical one with a bridge. There are various
> ways to achieve this.

Yes, but then those "workarounds" expose the given physical device to
the root namespace, which is exactly what I try to avoid here.  The
interface has no purpose outside of the specific target namespace, and
under no circumstances should the various applications on a normal Linux
system (whether it's network manager or whatever else) start to use the
device.  The same also applies to the kernel itself.  It is not
desirable to have the "root netns" start to do things like ipv6
stateless autoconfiguration, etc.

I of course know that all of those things can be individually disabled.

I just think having a physical netdev inside "single application"
namespaces is more complicated than it could be.

However, I have sufficiently made my argument clear, and I understand
that you don't share my concern.  This is perfectly fine. We agree to
disagree :)

I simply have to find the least intrusive work-around to my liking for
the intentional but so far undocumented behavior of netdevices vanishing
into thin air.  I'll manage.

-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: loosing netdevices with namespaces and unshare?

[Eric W. Biederman]

Cong Wang <xiyou.wangcong@gmail.com> writes:
> Network namespace does not special-case the physical devices,
> it treats them all equally as abstract net devices.

Absolutely not true.

The relevant code is in net/core/dev.c:default_device_exit

If a network device does not implement rntl_link_ops it is returned to
the initial network namespace.   Anything else will loose physical
devices.

Only for pure software based devices do we delete them.  Perhaps your
sub interface implements rtnl_link_ops?  Either that or something is
still holding a reference to your network namespace, which would prevent
the network device from being returned.

Eric

Re: loosing netdevices with namespaces and unshare?

[Harald Welte]

Hi Eric,

On Thu, Jun 01, 2017 at 01:32:49AM -0500, Eric W. Biederman wrote:

> If a network device does not implement rntl_link_ops it is returned to
> the initial network namespace.   Anything else will loose physical
> devices.

Thanks a lot for your statement.  This is a big relief, my line of
thinking thus is confirmed:  We shall not loose physical devices.

> Only for pure software based devices do we delete them.  Perhaps your
> sub interface implements rtnl_link_ops?  Either that or something is
> still holding a reference to your network namespace, which would prevent
> the network device from being returned.

My question is how to debug this further?  Monitoring
/proc/*/ns/net* showed that the ID of the namespace is gone after
terminating my processes in the namespace.  Short of adding printk() or
playing with kprobes: to the related kernel code, how can I track the
reference count or get an idea who might hold references?

Regards,
	Harald
-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: loosing netdevices with namespaces and unshare?

[Eric W. Biederman]

Harald Welte <laforge@gnumonks.org> writes:

> Hi Eric,
>
> On Thu, Jun 01, 2017 at 01:32:49AM -0500, Eric W. Biederman wrote:
>
>> If a network device does not implement rntl_link_ops it is returned to
>> the initial network namespace.   Anything else will loose physical
>> devices.
>
> Thanks a lot for your statement.  This is a big relief, my line of
> thinking thus is confirmed:  We shall not loose physical devices.

Rereading that I should have said:
    We shall not lose physical devices.
We should let the loose to talk and say interesting things to the world.

>> Only for pure software based devices do we delete them.  Perhaps your
>> sub interface implements rtnl_link_ops?  Either that or something is
>> still holding a reference to your network namespace, which would prevent
>> the network device from being returned.
>
> My question is how to debug this further?  Monitoring
> /proc/*/ns/net* showed that the ID of the namespace is gone after
> terminating my processes in the namespace.  Short of adding printk() or
> playing with kprobes: to the related kernel code, how can I track the
> reference count or get an idea who might hold references?

You mentioned sub-interface.  I would first look to see if your
sub-interface might possibly implement rtnl_link_ops.

For testing I would toss in a full fledged physical interface and
see if that pops back.  Just to verify what you are seeing happening is
happening.

In your minimal test case of "unshare -Urn bash -c 'sleep 1; exit 0;'" I
can't imagine there is anything holding a reference.  So it may come
down to adding some printks or playing with kprobes.

All of macvlans and vlans and anything I can think of as sub-interface
all implement rtnl_link_ops and will get deleted when a network
namespace exits.  Which generally is what you want as it gives a very
nice cleanup.

Eric

Re: loosing netdevices with namespaces and unshare?

[Cong Wang]

On Wed, May 31, 2017 at 11:32 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Cong Wang <xiyou.wangcong@gmail.com> writes:
>> Network namespace does not special-case the physical devices,
>> it treats them all equally as abstract net devices.
>
> Absolutely not true.
>
> The relevant code is in net/core/dev.c:default_device_exit
>
> If a network device does not implement rntl_link_ops it is returned to
> the initial network namespace.   Anything else will loose physical
> devices.

Hmm, I never noticed that if check...

>
> Only for pure software based devices do we delete them.  Perhaps your
> sub interface implements rtnl_link_ops?  Either that or something is
> still holding a reference to your network namespace, which would prevent
> the network device from being returned.
>

But this simply sucks:

                snprintf(fb_name, IFNAMSIZ, "dev%d", dev->ifindex);
                err = dev_change_net_namespace(dev, &init_net, fb_name);
                if (err) {
                        pr_emerg("%s: failed to move %s to init_net: %d\n",
                                 __func__, dev->name, err);
                        BUG();
                }

It is essentially hard to handle the error here, but it is quite easy to
trigger such BUG() by naming other device devX, it is no better
than just losing it.

Re: loosing netdevices with namespaces and unshare?

[Eric W. Biederman]

Cong Wang <xiyou.wangcong@gmail.com> writes:

> On Wed, May 31, 2017 at 11:32 PM, Eric W. Biederman
> <ebiederm@xmission.com> wrote:
>> Cong Wang <xiyou.wangcong@gmail.com> writes:
>>> Network namespace does not special-case the physical devices,
>>> it treats them all equally as abstract net devices.
>>
>> Absolutely not true.
>>
>> The relevant code is in net/core/dev.c:default_device_exit
>>
>> If a network device does not implement rntl_link_ops it is returned to
>> the initial network namespace.   Anything else will loose physical
>> devices.
>
> Hmm, I never noticed that if check...
>
>>
>> Only for pure software based devices do we delete them.  Perhaps your
>> sub interface implements rtnl_link_ops?  Either that or something is
>> still holding a reference to your network namespace, which would prevent
>> the network device from being returned.
>>
>
> But this simply sucks:
>
>                 snprintf(fb_name, IFNAMSIZ, "dev%d", dev->ifindex);
>                 err = dev_change_net_namespace(dev, &init_net, fb_name);
>                 if (err) {
>                         pr_emerg("%s: failed to move %s to init_net: %d\n",
>                                  __func__, dev->name, err);
>                         BUG();
>                 }
>
> It is essentially hard to handle the error here, but it is quite easy to
> trigger such BUG() by naming other device devX, it is no better
> than just losing it.

The rename only happens if there is a conflicting device name.

Beyond that there is the entire hotplug functionality so it should be
possible to automatically detect a new device in your network namespace
and do something with it.

Eric

Re: loosing netdevices with namespaces and unshare?

[David Ahern]

On 5/30/17 4:07 PM, Harald Welte wrote:
> In case you're wondering what I'm actually trying to achieve: Find
> an easy way to run a single program in an isolated namespace that only
> has one physical (usb) ethernet device.  I would like to execute that
> program as unprivileged user but still be able to bind to privileged
> ports.  And I want to do this using simple command-line tools without
> all the bloat and overhead of "container" solutions that have 99% of
> features I don't need.  But let that not distract you, I think the
> mysteriously disappearing netdevices are a more general and important
> issue.

An alternative approach is to create a bridge and add the usb ethernet
device to it. As you want to launch a program, create a veth pair. Put
one end into the bridge, and the other end into the new network namespace.

All of this can be scripted quite easily with 'ip' - including
configuring the veth device pushed into the namespace and running the
command. Use unshare for the other namespaces.