unregister_netdevice warnings when deleting netns

unregister_netdevice warnings when deleting netns

[Anand Gurram]

Hi,

I am currently using kernel version 3.16.7 on a linux switch.
While creating and destroying network namespaces I am observing below logs
on the console
"unregister_netdevice: waiting for lo to become free. Usage count = 1"

Can you please suggest and provide instructions on how to debug this issue.
If any fix already available can you please point me to the link.

Best Regards,
Anand

Re: unregister_netdevice warnings when deleting netns

[Julian Anastasov]

	Hello,

On Mon, 28 Sep 2015, Anand Gurram wrote:

> I am currently using kernel version 3.16.7 on a linux switch.
> While creating and destroying network namespaces I am observing below logs
> on the console
> "unregister_netdevice: waiting for lo to become free. Usage count = 1"
> 
> Can you please suggest and provide instructions on how to debug this issue.
> If any fix already available can you please point me to the link.

	There are two commits from Linux 4.2 that may help:

commit e9e4dd3267d0 ("net: do not process device backlog during unregistration")
commit 2c17d27c36dc ("net: call rcu_read_lock early in process_backlog")

	For now I see them only in 3.2.71+ and 3.12.48+.
I think, they will appear in other stable versions too...

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: unregister_netdevice warnings when deleting netns

[Eric W. Biederman]

Julian Anastasov <ja@ssi.bg> writes:

> 	Hello,
>
> On Mon, 28 Sep 2015, Anand Gurram wrote:
>
>> I am currently using kernel version 3.16.7 on a linux switch.
>> While creating and destroying network namespaces I am observing below logs
>> on the console
>> "unregister_netdevice: waiting for lo to become free. Usage count = 1"
>> 
>> Can you please suggest and provide instructions on how to debug this issue.
>> If any fix already available can you please point me to the link.
>
> 	There are two commits from Linux 4.2 that may help:
>
> commit e9e4dd3267d0 ("net: do not process device backlog during unregistration")
> commit 2c17d27c36dc ("net: call rcu_read_lock early in process_backlog")
>
> 	For now I see them only in 3.2.71+ and 3.12.48+.
> I think, they will appear in other stable versions too...

If that message repeats indefinitely it means there is a leaked
reference to the network namespaces lo device.

If the message just spits out a few times and then goes away it simply
means that something is taking a while to cleanup and drop it's
reference.

This is slightly complicated by the fact that it is not uncommon when a
network device goes away to redirect all references to itself to the lo
device.

Eric

Re: unregister_netdevice warnings when deleting netns

[Julian Anastasov]

	Hello,

On Mon, 28 Sep 2015, Eric W. Biederman wrote:

> Julian Anastasov <ja@ssi.bg> writes:
> 
> > On Mon, 28 Sep 2015, Anand Gurram wrote:
> >
> >> I am currently using kernel version 3.16.7 on a linux switch.
> >> While creating and destroying network namespaces I am observing below logs
> >> on the console
> >> "unregister_netdevice: waiting for lo to become free. Usage count = 1"
> >> 
> >> Can you please suggest and provide instructions on how to debug this issue.
> >> If any fix already available can you please point me to the link.
> >
> > 	There are two commits from Linux 4.2 that may help:
> >
> > commit e9e4dd3267d0 ("net: do not process device backlog during unregistration")
> > commit 2c17d27c36dc ("net: call rcu_read_lock early in process_backlog")
> >
> If that message repeats indefinitely it means there is a leaked
> reference to the network namespaces lo device.
> 
> If the message just spits out a few times and then goes away it simply
> means that something is taking a while to cleanup and drop it's
> reference.
> 
> This is slightly complicated by the fact that it is not uncommon when a
> network device goes away to redirect all references to itself to the lo
> device.

	Yes, there is a little chance with forwarding disabled,
i.e. when presence of "ipv4: Avoid crashing in ip_error" does
not matter, flying packet to leave new reference somewhere,
without crashing. But it may be another problem, of course.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: unregister_netdevice warnings when deleting netns

[Anand Gurram]

Thanks Julian, I will check if these two commits work for me.

>I think, they will appear in other stable versions too...

Yes, I saw them in other versions, the fix which is suggested in those
branches didn't work for me.
Hope the above two commits help.

Regards,
Anand

On Tue, Sep 29, 2015 at 12:42 AM, Julian Anastasov <ja@ssi.bg> wrote:
>
>         Hello,
>
> On Mon, 28 Sep 2015, Anand Gurram wrote:
>
>> I am currently using kernel version 3.16.7 on a linux switch.
>> While creating and destroying network namespaces I am observing below logs
>> on the console
>> "unregister_netdevice: waiting for lo to become free. Usage count = 1"
>>
>> Can you please suggest and provide instructions on how to debug this issue.
>> If any fix already available can you please point me to the link.
>
>         There are two commits from Linux 4.2 that may help:
>
> commit e9e4dd3267d0 ("net: do not process device backlog during unregistration")
> commit 2c17d27c36dc ("net: call rcu_read_lock early in process_backlog")
>
>         For now I see them only in 3.2.71+ and 3.12.48+.
> I think, they will appear in other stable versions too...
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>

Re: unregister_netdevice warnings when deleting netns

[Anand Gurram]

>If the message just spits out a few times and then goes away it simply
>means that something is taking a while to cleanup and drop it's
>reference.

The message just spits out few times and then goes away, I am trying
to debug why cleanup is taking long,
and where it is still referenced. Any pointers in debugging such
issues will be of great help.

Best Regards,
Anand

On Tue, Sep 29, 2015 at 3:05 AM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Julian Anastasov <ja@ssi.bg> writes:
>
>>       Hello,
>>
>> On Mon, 28 Sep 2015, Anand Gurram wrote:
>>
>>> I am currently using kernel version 3.16.7 on a linux switch.
>>> While creating and destroying network namespaces I am observing below logs
>>> on the console
>>> "unregister_netdevice: waiting for lo to become free. Usage count = 1"
>>>
>>> Can you please suggest and provide instructions on how to debug this issue.
>>> If any fix already available can you please point me to the link.
>>
>>       There are two commits from Linux 4.2 that may help:
>>
>> commit e9e4dd3267d0 ("net: do not process device backlog during unregistration")
>> commit 2c17d27c36dc ("net: call rcu_read_lock early in process_backlog")
>>
>>       For now I see them only in 3.2.71+ and 3.12.48+.
>> I think, they will appear in other stable versions too...
>
> If that message repeats indefinitely it means there is a leaked
> reference to the network namespaces lo device.
>
> If the message just spits out a few times and then goes away it simply
> means that something is taking a while to cleanup and drop it's
> reference.
>
> This is slightly complicated by the fact that it is not uncommon when a
> network device goes away to redirect all references to itself to the lo
> device.
>
> Eric

Re: unregister_netdevice warnings when deleting netns

[Eric W. Biederman]

Anand Gurram <anandarao.gurram@gmail.com> writes:

>>If the message just spits out a few times and then goes away it simply
>>means that something is taking a while to cleanup and drop it's
>>reference.
>
> The message just spits out few times and then goes away, I am trying
> to debug why cleanup is taking long,
> and where it is still referenced. Any pointers in debugging such
> issues will be of great help.

The one thing I have done in the past is to instrument dev_hold
and dev_put and look where in the code the stragglers are coming from
(when I can reproduce the issue reliably).

Sometimes people have addressed this class of issue with code review,
but with a slow cleanup you can't catch this by finding a missing
dev_put.

It takes some creativity to find these as people rarely make the same
mistake twice.

Eric

Re: unregister_netdevice warnings when deleting netns

[Anand Gurram]

Hi Julian and Eric

I tried both the patches which you have suggested, the issue is still
seen, I am observing same warning message thrown on the console
  "unregister_netdevice: waiting for lo to become free. Usage count = 1".


>Sometimes people have addressed this class of issue with code review,
>but with a slow cleanup you can't catch this by finding a missing
>dev_put.

Yeah, currently since slow cleanup is happening I am unable to trace
just by having count for dev_hold and dev_put.


Actually at the time of tearing down the name space there is an active
TCP connection present.
When this TCP connection is not present then we are not seeing this issue.

Any additional ideas and suggestions on debugging in above scenario?

Best Regards,
Anand

On Tue, Sep 29, 2015 at 12:14 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Anand Gurram <anandarao.gurram@gmail.com> writes:
>
>>>If the message just spits out a few times and then goes away it simply
>>>means that something is taking a while to cleanup and drop it's
>>>reference.
>>
>> The message just spits out few times and then goes away, I am trying
>> to debug why cleanup is taking long,
>> and where it is still referenced. Any pointers in debugging such
>> issues will be of great help.
>
> The one thing I have done in the past is to instrument dev_hold
> and dev_put and look where in the code the stragglers are coming from
> (when I can reproduce the issue reliably).
>
> Sometimes people have addressed this class of issue with code review,
> but with a slow cleanup you can't catch this by finding a missing
> dev_put.
>
> It takes some creativity to find these as people rarely make the same
> mistake twice.
>
> Eric

Re: unregister_netdevice warnings when deleting netns

[William Dauchy]

On Tue, Sep 29, 2015 at 6:12 PM, Anand Gurram
<anandarao.gurram@gmail.com> wrote:
> Any additional ideas and suggestions on debugging in above scenario?

You may give a try with the recent patch:
packet: race condition in packet_bind
http://www.spinics.net/lists/netdev/msg351337.html

-- 
William