* [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet
@ 2020-06-30 22:18 Martin KaFai Lau
2020-06-30 23:20 ` Willem de Bruijn
0 siblings, 1 reply; 5+ messages in thread
From: Martin KaFai Lau @ 2020-06-30 22:18 UTC (permalink / raw)
To: netdev; +Cc: David Miller, kernel-team, Willem de Bruijn
When testing a recent kernel (5.6 in our case), the skb->mark of the
IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is
discovered by the bpf@tc that depends on skb->mark to work properly.
The same bpf prog has been working in the earlier kernel version.
After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"),
the skb->mark is set and seen by bpf@tc properly.
We have noticed that in IPv4 TCP RST but it should also
happen to the ACK based on tcp_v4_send_ack() is also depending
on ip_send_unicast_reply().
This patch tries to fix it by initializing the ipc.sockc.mark to
fl4.flowi4_mark.
Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg")
Cc: Willem de Bruijn <willemb@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
net/ipv4/ip_output.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 090d3097ee15..033512f719ec 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb,
sk->sk_bound_dev_if = arg->bound_dev_if;
sk->sk_sndbuf = sysctl_wmem_default;
sk->sk_mark = fl4.flowi4_mark;
+ ipc.sockc.mark = fl4.flowi4_mark;
err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
len, 0, &ipc, &rt, MSG_DONTWAIT);
if (unlikely(err)) {
--
2.24.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet
2020-06-30 22:18 [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet Martin KaFai Lau
@ 2020-06-30 23:20 ` Willem de Bruijn
2020-06-30 23:45 ` Martin KaFai Lau
0 siblings, 1 reply; 5+ messages in thread
From: Willem de Bruijn @ 2020-06-30 23:20 UTC (permalink / raw)
To: Martin KaFai Lau; +Cc: Network Development, David Miller, Kernel Team
On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote:
>
> When testing a recent kernel (5.6 in our case), the skb->mark of the
> IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is
> discovered by the bpf@tc that depends on skb->mark to work properly.
> The same bpf prog has been working in the earlier kernel version.
> After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"),
> the skb->mark is set and seen by bpf@tc properly.
>
> We have noticed that in IPv4 TCP RST but it should also
> happen to the ACK based on tcp_v4_send_ack() is also depending
> on ip_send_unicast_reply().
>
> This patch tries to fix it by initializing the ipc.sockc.mark to
> fl4.flowi4_mark.
>
> Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg")
> Cc: Willem de Bruijn <willemb@google.com>
> Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> ---
> net/ipv4/ip_output.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
> index 090d3097ee15..033512f719ec 100644
> --- a/net/ipv4/ip_output.c
> +++ b/net/ipv4/ip_output.c
> @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb,
> sk->sk_bound_dev_if = arg->bound_dev_if;
> sk->sk_sndbuf = sysctl_wmem_default;
> sk->sk_mark = fl4.flowi4_mark;
> + ipc.sockc.mark = fl4.flowi4_mark;
> err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
> len, 0, &ipc, &rt, MSG_DONTWAIT);
> if (unlikely(err)) {
Yes, this total sense. I missed these cases.
Slight modification, the line above then no longer needs to be set.
That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation
with fwmark_reflect enabled"). Basically, it pretends that the socket
has a mark associated, but sk here is always the (netns) global
control sock. So your BPF program was depending on fwmark_reflect?
ipv6 seems to work differently enough not to have this problem,
tcp_v6_send_response passing fl6.flowi6_mark directly to ip6_xmit.
This was added in commit commit 92e55f412cff ("tcp: don't annotate
mark on control socket from tcp_v6_send_response()").
But I do see the same pattern where a socket mark is set from a
reflected value in icmp_reply and __icmp_send. Those almost certainly
need updating too. I can do that separately if you prefer. I even
placed ipcm_init right below this sk_mark initialization without
considering ipcm_init_sk. D'oh.
sk->sk_mark = mark;
ipcm_init(&ipc);
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet
2020-06-30 23:20 ` Willem de Bruijn
@ 2020-06-30 23:45 ` Martin KaFai Lau
2020-07-01 0:45 ` Willem de Bruijn
0 siblings, 1 reply; 5+ messages in thread
From: Martin KaFai Lau @ 2020-06-30 23:45 UTC (permalink / raw)
To: Willem de Bruijn; +Cc: Network Development, David Miller, Kernel Team
On Tue, Jun 30, 2020 at 07:20:46PM -0400, Willem de Bruijn wrote:
> On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote:
> >
> > When testing a recent kernel (5.6 in our case), the skb->mark of the
> > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is
> > discovered by the bpf@tc that depends on skb->mark to work properly.
> > The same bpf prog has been working in the earlier kernel version.
> > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"),
> > the skb->mark is set and seen by bpf@tc properly.
> >
> > We have noticed that in IPv4 TCP RST but it should also
> > happen to the ACK based on tcp_v4_send_ack() is also depending
> > on ip_send_unicast_reply().
> >
> > This patch tries to fix it by initializing the ipc.sockc.mark to
> > fl4.flowi4_mark.
> >
> > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg")
> > Cc: Willem de Bruijn <willemb@google.com>
> > Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> > ---
> > net/ipv4/ip_output.c | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
> > index 090d3097ee15..033512f719ec 100644
> > --- a/net/ipv4/ip_output.c
> > +++ b/net/ipv4/ip_output.c
> > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb,
> > sk->sk_bound_dev_if = arg->bound_dev_if;
> > sk->sk_sndbuf = sysctl_wmem_default;
> > sk->sk_mark = fl4.flowi4_mark;
> > + ipc.sockc.mark = fl4.flowi4_mark;
> > err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
> > len, 0, &ipc, &rt, MSG_DONTWAIT);
> > if (unlikely(err)) {
>
> Yes, this total sense. I missed these cases.
>
> Slight modification, the line above then no longer needs to be set.
> That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation
> with fwmark_reflect enabled"). Basically, it pretends that the socket
> has a mark associated, but sk here is always the (netns) global
> control sock. So your BPF program was depending on fwmark_reflect?
Make sense. I was also tempting to remove the line above.
Thanks for the commit pointer.
No, the BPF program does not depend on fwmark_reflect. It depends
on the sk->sk_mark set by a user space process.
I was also considering to do ipcm_init_sk() but then rolled back
because of the global control sock here.
>
> ipv6 seems to work differently enough not to have this problem,
> tcp_v6_send_response passing fl6.flowi6_mark directly to ip6_xmit.
> This was added in commit commit 92e55f412cff ("tcp: don't annotate
> mark on control socket from tcp_v6_send_response()").
Correct. IPv6 does it differently, so the same problem is
not observed in IPv6.
>
> But I do see the same pattern where a socket mark is set from a
> reflected value in icmp_reply and __icmp_send. Those almost certainly
> need updating too. I can do that separately if you prefer. I even
> placed ipcm_init right below this sk_mark initialization without
> considering ipcm_init_sk. D'oh.
Good point. I think it will only be a few lines change altogether,
so it makes little sense to break up the fix. I will toss mine and
wait for yours ;)
Thanks for your help!
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet
2020-06-30 23:45 ` Martin KaFai Lau
@ 2020-07-01 0:45 ` Willem de Bruijn
2020-07-01 1:22 ` Martin KaFai Lau
0 siblings, 1 reply; 5+ messages in thread
From: Willem de Bruijn @ 2020-07-01 0:45 UTC (permalink / raw)
To: Martin KaFai Lau
Cc: Willem de Bruijn, Network Development, David Miller, Kernel Team
On Tue, Jun 30, 2020 at 7:46 PM Martin KaFai Lau <kafai@fb.com> wrote:
>
> On Tue, Jun 30, 2020 at 07:20:46PM -0400, Willem de Bruijn wrote:
> > On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote:
> > >
> > > When testing a recent kernel (5.6 in our case), the skb->mark of the
> > > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is
> > > discovered by the bpf@tc that depends on skb->mark to work properly.
> > > The same bpf prog has been working in the earlier kernel version.
> > > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"),
> > > the skb->mark is set and seen by bpf@tc properly.
> > >
> > > We have noticed that in IPv4 TCP RST but it should also
> > > happen to the ACK based on tcp_v4_send_ack() is also depending
> > > on ip_send_unicast_reply().
> > >
> > > This patch tries to fix it by initializing the ipc.sockc.mark to
> > > fl4.flowi4_mark.
> > >
> > > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg")
> > > Cc: Willem de Bruijn <willemb@google.com>
> > > Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> > > ---
> > > net/ipv4/ip_output.c | 1 +
> > > 1 file changed, 1 insertion(+)
> > >
> > > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
> > > index 090d3097ee15..033512f719ec 100644
> > > --- a/net/ipv4/ip_output.c
> > > +++ b/net/ipv4/ip_output.c
> > > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb,
> > > sk->sk_bound_dev_if = arg->bound_dev_if;
> > > sk->sk_sndbuf = sysctl_wmem_default;
> > > sk->sk_mark = fl4.flowi4_mark;
> > > + ipc.sockc.mark = fl4.flowi4_mark;
> > > err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
> > > len, 0, &ipc, &rt, MSG_DONTWAIT);
> > > if (unlikely(err)) {
> >
> > Yes, this total sense. I missed these cases.
> >
> > Slight modification, the line above then no longer needs to be set.
> > That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation
> > with fwmark_reflect enabled"). Basically, it pretends that the socket
> > has a mark associated, but sk here is always the (netns) global
> > control sock. So your BPF program was depending on fwmark_reflect?
> Make sense. I was also tempting to remove the line above.
> Thanks for the commit pointer.
>
> No, the BPF program does not depend on fwmark_reflect. It depends
> on the sk->sk_mark set by a user space process.
Then I don't fully understand, as ip_send_unicast_reply is only called
with the per-netns percpu ctl_sk.
> I was also considering to do ipcm_init_sk() but then rolled back
> because of the global control sock here.
>
> >
> > ipv6 seems to work differently enough not to have this problem,
> > tcp_v6_send_response passing fl6.flowi6_mark directly to ip6_xmit.
> > This was added in commit commit 92e55f412cff ("tcp: don't annotate
> > mark on control socket from tcp_v6_send_response()").
> Correct. IPv6 does it differently, so the same problem is
> not observed in IPv6.
>
> >
> > But I do see the same pattern where a socket mark is set from a
> > reflected value in icmp_reply and __icmp_send. Those almost certainly
> > need updating too. I can do that separately if you prefer. I even
> > placed ipcm_init right below this sk_mark initialization without
> > considering ipcm_init_sk. D'oh.
> Good point. I think it will only be a few lines change altogether,
> so it makes little sense to break up the fix. I will toss mine and
> wait for yours ;)
Will do. Want to double check my initial rushed reading first.
> Thanks for your help!
Not at all. Apologies for the breakage.. Thanks for the initial fix!
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet
2020-07-01 0:45 ` Willem de Bruijn
@ 2020-07-01 1:22 ` Martin KaFai Lau
0 siblings, 0 replies; 5+ messages in thread
From: Martin KaFai Lau @ 2020-07-01 1:22 UTC (permalink / raw)
To: Willem de Bruijn; +Cc: Network Development, David Miller, Kernel Team
On Tue, Jun 30, 2020 at 08:45:13PM -0400, Willem de Bruijn wrote:
> On Tue, Jun 30, 2020 at 7:46 PM Martin KaFai Lau <kafai@fb.com> wrote:
> >
> > On Tue, Jun 30, 2020 at 07:20:46PM -0400, Willem de Bruijn wrote:
> > > On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote:
> > > >
> > > > When testing a recent kernel (5.6 in our case), the skb->mark of the
> > > > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is
> > > > discovered by the bpf@tc that depends on skb->mark to work properly.
> > > > The same bpf prog has been working in the earlier kernel version.
> > > > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"),
> > > > the skb->mark is set and seen by bpf@tc properly.
> > > >
> > > > We have noticed that in IPv4 TCP RST but it should also
> > > > happen to the ACK based on tcp_v4_send_ack() is also depending
> > > > on ip_send_unicast_reply().
> > > >
> > > > This patch tries to fix it by initializing the ipc.sockc.mark to
> > > > fl4.flowi4_mark.
> > > >
> > > > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg")
> > > > Cc: Willem de Bruijn <willemb@google.com>
> > > > Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> > > > ---
> > > > net/ipv4/ip_output.c | 1 +
> > > > 1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
> > > > index 090d3097ee15..033512f719ec 100644
> > > > --- a/net/ipv4/ip_output.c
> > > > +++ b/net/ipv4/ip_output.c
> > > > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb,
> > > > sk->sk_bound_dev_if = arg->bound_dev_if;
> > > > sk->sk_sndbuf = sysctl_wmem_default;
> > > > sk->sk_mark = fl4.flowi4_mark;
> > > > + ipc.sockc.mark = fl4.flowi4_mark;
> > > > err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
> > > > len, 0, &ipc, &rt, MSG_DONTWAIT);
> > > > if (unlikely(err)) {
> > >
> > > Yes, this total sense. I missed these cases.
> > >
> > > Slight modification, the line above then no longer needs to be set.
> > > That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation
> > > with fwmark_reflect enabled"). Basically, it pretends that the socket
> > > has a mark associated, but sk here is always the (netns) global
> > > control sock. So your BPF program was depending on fwmark_reflect?
> > Make sense. I was also tempting to remove the line above.
> > Thanks for the commit pointer.
> >
> > No, the BPF program does not depend on fwmark_reflect. It depends
> > on the sk->sk_mark set by a user space process.
>
> Then I don't fully understand, as ip_send_unicast_reply is only called
> with the per-netns percpu ctl_sk.
Before this proposed patch, my understanding is,
the ctl_sk->sk_mark is correctly set here in ip_send_unicast_reply().
The ctl_sk->sk_mark was actually set earlier in the tcp_v4_send_reset().
However, ctl_sk->sk_mark is not used to set the skb->mark.
Instead, cork->mark is now used to initialize the skb->mark
in __ipmake_skb().
The cork->mark is not properly set in ip_setup_cork()
because ipc->sockc.mark is 0 here which this proposed patch
is trying to address.
The call stack is something like this:
tcp_v4_send_reset()
ctl_sk->sk_mark = (sk->sk_state == TCP_TIME_WAIT) ?... /* <- sk_mark is set */
=> ip_send_unicast_reply()
=> ip_append_data(..., &ipc, ...);
=> ip_setup_cork(..., &inet->cork.base, ipc, ...);
cork->mark = ipc->sockc.mark; /* <-- ipc->sockc.mark is 0 */
=> ip_push_pending_frames(sk, &fl4);
=> ip_finish_skb(sk, fl4)
=> __ip_make_skb(sk, fl4, ..., &inet_sk(sk)->cork.base);
skb->mark = cork->mark; /* <-- cork->mark is 0 */
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-07-01 1:22 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-30 22:18 [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet Martin KaFai Lau
2020-06-30 23:20 ` Willem de Bruijn
2020-06-30 23:45 ` Martin KaFai Lau
2020-07-01 0:45 ` Willem de Bruijn
2020-07-01 1:22 ` Martin KaFai Lau
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.