BUG: TCP timewait sockets survive across namespace creation in net-next

BUG: TCP timewait sockets survive across namespace creation in net-next

[Leonard Crestez]

Hello,

It appears that in recent net-next versions it is possible for sockets 
in the timewait state to survive across namespace add/del. Timewait 
sockets are inserted into a global hash and only the sock_net value is 
compared when they are enumerated from interfaces like /proc/net/tcp and 
inet_diag. Old TW sockets are not cleared after namespace delete and 
namespaces are allocated from a slab and thus their pointers get reused 
a lot, when that happens timewait sockets from an old namespace will 
show up in the new one.

This can be reproduced by establishing a TCP connection over a veth pair 
between two namespaces, closing and then recreating those namespaces. 
Old timewait sockets will be visible and it happens quite reliably, 
often on the first iteration. I can try to provide a script for this.

I can't point to specific bugs outside of tests that explicitly 
enumerate timewait sockets but letting sk_net be a dangling pointer 
seems very dangerous. It also violates the idea of network namespaces 
being independent and isolated.

This does not happen in 5.17, I bisected this behavior to commit 
0dad4087a86a ("tcp/dccp: get rid of inet_twsk_purge()")

--
Regards,
Leonard

Re: BUG: TCP timewait sockets survive across namespace creation in net-next

[Eric Dumazet]

On Thu, May 12, 2022 at 11:01 AM Leonard Crestez <cdleonard@gmail.com> wrote:
>
> Hello,
>
> It appears that in recent net-next versions it is possible for sockets
> in the timewait state to survive across namespace add/del. Timewait
> sockets are inserted into a global hash and only the sock_net value is
> compared when they are enumerated from interfaces like /proc/net/tcp and
> inet_diag. Old TW sockets are not cleared after namespace delete and
> namespaces are allocated from a slab and thus their pointers get reused
> a lot, when that happens timewait sockets from an old namespace will
> show up in the new one.
>
> This can be reproduced by establishing a TCP connection over a veth pair
> between two namespaces, closing and then recreating those namespaces.
> Old timewait sockets will be visible and it happens quite reliably,
> often on the first iteration. I can try to provide a script for this.
>
> I can't point to specific bugs outside of tests that explicitly
> enumerate timewait sockets but letting sk_net be a dangling pointer
> seems very dangerous. It also violates the idea of network namespaces
> being independent and isolated.
>
> This does not happen in 5.17, I bisected this behavior to commit
> 0dad4087a86a ("tcp/dccp: get rid of inet_twsk_purge()")
>

Thanks for the report.

I guess we will need to store the (struct net)->net_cookie to
disambiguate the case
where a new 'struct net' is reusing the same storage than an old one.

Re: BUG: TCP timewait sockets survive across namespace creation in net-next

[Eric Dumazet]

On Thu, May 12, 2022 at 11:13 AM Eric Dumazet <edumazet@google.com> wrote:
>
> On Thu, May 12, 2022 at 11:01 AM Leonard Crestez <cdleonard@gmail.com> wrote:
> >
> > Hello,
> >
> > It appears that in recent net-next versions it is possible for sockets
> > in the timewait state to survive across namespace add/del. Timewait
> > sockets are inserted into a global hash and only the sock_net value is
> > compared when they are enumerated from interfaces like /proc/net/tcp and
> > inet_diag. Old TW sockets are not cleared after namespace delete and
> > namespaces are allocated from a slab and thus their pointers get reused
> > a lot, when that happens timewait sockets from an old namespace will
> > show up in the new one.
> >
> > This can be reproduced by establishing a TCP connection over a veth pair
> > between two namespaces, closing and then recreating those namespaces.
> > Old timewait sockets will be visible and it happens quite reliably,
> > often on the first iteration. I can try to provide a script for this.
> >
> > I can't point to specific bugs outside of tests that explicitly
> > enumerate timewait sockets but letting sk_net be a dangling pointer
> > seems very dangerous. It also violates the idea of network namespaces
> > being independent and isolated.
> >
> > This does not happen in 5.17, I bisected this behavior to commit
> > 0dad4087a86a ("tcp/dccp: get rid of inet_twsk_purge()")
> >
>
> Thanks for the report.
>
> I guess we will need to store the (struct net)->net_cookie to
> disambiguate the case
> where a new 'struct net' is reusing the same storage than an old one.

Oh well, too many changes would be needed.
I will send a revert, thanks.