* [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet
@ 2020-06-30 22:18 Martin KaFai Lau
2020-06-30 23:20 ` Willem de Bruijn
0 siblings, 1 reply; 5+ messages in thread
From: Martin KaFai Lau @ 2020-06-30 22:18 UTC (permalink / raw)
To: netdev; +Cc: David Miller, kernel-team, Willem de Bruijn
When testing a recent kernel (5.6 in our case), the skb->mark of the
IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is
discovered by the bpf@tc that depends on skb->mark to work properly.
The same bpf prog has been working in the earlier kernel version.
After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"),
the skb->mark is set and seen by bpf@tc properly.
We have noticed that in IPv4 TCP RST but it should also
happen to the ACK based on tcp_v4_send_ack() is also depending
on ip_send_unicast_reply().
This patch tries to fix it by initializing the ipc.sockc.mark to
fl4.flowi4_mark.
Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg")
Cc: Willem de Bruijn <willemb@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
net/ipv4/ip_output.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 090d3097ee15..033512f719ec 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb,
sk->sk_bound_dev_if = arg->bound_dev_if;
sk->sk_sndbuf = sysctl_wmem_default;
sk->sk_mark = fl4.flowi4_mark;
+ ipc.sockc.mark = fl4.flowi4_mark;
err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
len, 0, &ipc, &rt, MSG_DONTWAIT);
if (unlikely(err)) {
--
2.24.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet 2020-06-30 22:18 [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet Martin KaFai Lau @ 2020-06-30 23:20 ` Willem de Bruijn 2020-06-30 23:45 ` Martin KaFai Lau 0 siblings, 1 reply; 5+ messages in thread From: Willem de Bruijn @ 2020-06-30 23:20 UTC (permalink / raw) To: Martin KaFai Lau; +Cc: Network Development, David Miller, Kernel Team On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote: > > When testing a recent kernel (5.6 in our case), the skb->mark of the > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is > discovered by the bpf@tc that depends on skb->mark to work properly. > The same bpf prog has been working in the earlier kernel version. > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"), > the skb->mark is set and seen by bpf@tc properly. > > We have noticed that in IPv4 TCP RST but it should also > happen to the ACK based on tcp_v4_send_ack() is also depending > on ip_send_unicast_reply(). > > This patch tries to fix it by initializing the ipc.sockc.mark to > fl4.flowi4_mark. > > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg") > Cc: Willem de Bruijn <willemb@google.com> > Signed-off-by: Martin KaFai Lau <kafai@fb.com> > --- > net/ipv4/ip_output.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c > index 090d3097ee15..033512f719ec 100644 > --- a/net/ipv4/ip_output.c > +++ b/net/ipv4/ip_output.c > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb, > sk->sk_bound_dev_if = arg->bound_dev_if; > sk->sk_sndbuf = sysctl_wmem_default; > sk->sk_mark = fl4.flowi4_mark; > + ipc.sockc.mark = fl4.flowi4_mark; > err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base, > len, 0, &ipc, &rt, MSG_DONTWAIT); > if (unlikely(err)) { Yes, this total sense. I missed these cases. Slight modification, the line above then no longer needs to be set. That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation with fwmark_reflect enabled"). Basically, it pretends that the socket has a mark associated, but sk here is always the (netns) global control sock. So your BPF program was depending on fwmark_reflect? ipv6 seems to work differently enough not to have this problem, tcp_v6_send_response passing fl6.flowi6_mark directly to ip6_xmit. This was added in commit commit 92e55f412cff ("tcp: don't annotate mark on control socket from tcp_v6_send_response()"). But I do see the same pattern where a socket mark is set from a reflected value in icmp_reply and __icmp_send. Those almost certainly need updating too. I can do that separately if you prefer. I even placed ipcm_init right below this sk_mark initialization without considering ipcm_init_sk. D'oh. sk->sk_mark = mark; ipcm_init(&ipc); ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet 2020-06-30 23:20 ` Willem de Bruijn @ 2020-06-30 23:45 ` Martin KaFai Lau 2020-07-01 0:45 ` Willem de Bruijn 0 siblings, 1 reply; 5+ messages in thread From: Martin KaFai Lau @ 2020-06-30 23:45 UTC (permalink / raw) To: Willem de Bruijn; +Cc: Network Development, David Miller, Kernel Team On Tue, Jun 30, 2020 at 07:20:46PM -0400, Willem de Bruijn wrote: > On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote: > > > > When testing a recent kernel (5.6 in our case), the skb->mark of the > > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is > > discovered by the bpf@tc that depends on skb->mark to work properly. > > The same bpf prog has been working in the earlier kernel version. > > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"), > > the skb->mark is set and seen by bpf@tc properly. > > > > We have noticed that in IPv4 TCP RST but it should also > > happen to the ACK based on tcp_v4_send_ack() is also depending > > on ip_send_unicast_reply(). > > > > This patch tries to fix it by initializing the ipc.sockc.mark to > > fl4.flowi4_mark. > > > > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg") > > Cc: Willem de Bruijn <willemb@google.com> > > Signed-off-by: Martin KaFai Lau <kafai@fb.com> > > --- > > net/ipv4/ip_output.c | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c > > index 090d3097ee15..033512f719ec 100644 > > --- a/net/ipv4/ip_output.c > > +++ b/net/ipv4/ip_output.c > > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb, > > sk->sk_bound_dev_if = arg->bound_dev_if; > > sk->sk_sndbuf = sysctl_wmem_default; > > sk->sk_mark = fl4.flowi4_mark; > > + ipc.sockc.mark = fl4.flowi4_mark; > > err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base, > > len, 0, &ipc, &rt, MSG_DONTWAIT); > > if (unlikely(err)) { > > Yes, this total sense. I missed these cases. > > Slight modification, the line above then no longer needs to be set. > That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation > with fwmark_reflect enabled"). Basically, it pretends that the socket > has a mark associated, but sk here is always the (netns) global > control sock. So your BPF program was depending on fwmark_reflect? Make sense. I was also tempting to remove the line above. Thanks for the commit pointer. No, the BPF program does not depend on fwmark_reflect. It depends on the sk->sk_mark set by a user space process. I was also considering to do ipcm_init_sk() but then rolled back because of the global control sock here. > > ipv6 seems to work differently enough not to have this problem, > tcp_v6_send_response passing fl6.flowi6_mark directly to ip6_xmit. > This was added in commit commit 92e55f412cff ("tcp: don't annotate > mark on control socket from tcp_v6_send_response()"). Correct. IPv6 does it differently, so the same problem is not observed in IPv6. > > But I do see the same pattern where a socket mark is set from a > reflected value in icmp_reply and __icmp_send. Those almost certainly > need updating too. I can do that separately if you prefer. I even > placed ipcm_init right below this sk_mark initialization without > considering ipcm_init_sk. D'oh. Good point. I think it will only be a few lines change altogether, so it makes little sense to break up the fix. I will toss mine and wait for yours ;) Thanks for your help! ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet 2020-06-30 23:45 ` Martin KaFai Lau @ 2020-07-01 0:45 ` Willem de Bruijn 2020-07-01 1:22 ` Martin KaFai Lau 0 siblings, 1 reply; 5+ messages in thread From: Willem de Bruijn @ 2020-07-01 0:45 UTC (permalink / raw) To: Martin KaFai Lau Cc: Willem de Bruijn, Network Development, David Miller, Kernel Team On Tue, Jun 30, 2020 at 7:46 PM Martin KaFai Lau <kafai@fb.com> wrote: > > On Tue, Jun 30, 2020 at 07:20:46PM -0400, Willem de Bruijn wrote: > > On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote: > > > > > > When testing a recent kernel (5.6 in our case), the skb->mark of the > > > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is > > > discovered by the bpf@tc that depends on skb->mark to work properly. > > > The same bpf prog has been working in the earlier kernel version. > > > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"), > > > the skb->mark is set and seen by bpf@tc properly. > > > > > > We have noticed that in IPv4 TCP RST but it should also > > > happen to the ACK based on tcp_v4_send_ack() is also depending > > > on ip_send_unicast_reply(). > > > > > > This patch tries to fix it by initializing the ipc.sockc.mark to > > > fl4.flowi4_mark. > > > > > > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg") > > > Cc: Willem de Bruijn <willemb@google.com> > > > Signed-off-by: Martin KaFai Lau <kafai@fb.com> > > > --- > > > net/ipv4/ip_output.c | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c > > > index 090d3097ee15..033512f719ec 100644 > > > --- a/net/ipv4/ip_output.c > > > +++ b/net/ipv4/ip_output.c > > > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb, > > > sk->sk_bound_dev_if = arg->bound_dev_if; > > > sk->sk_sndbuf = sysctl_wmem_default; > > > sk->sk_mark = fl4.flowi4_mark; > > > + ipc.sockc.mark = fl4.flowi4_mark; > > > err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base, > > > len, 0, &ipc, &rt, MSG_DONTWAIT); > > > if (unlikely(err)) { > > > > Yes, this total sense. I missed these cases. > > > > Slight modification, the line above then no longer needs to be set. > > That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation > > with fwmark_reflect enabled"). Basically, it pretends that the socket > > has a mark associated, but sk here is always the (netns) global > > control sock. So your BPF program was depending on fwmark_reflect? > Make sense. I was also tempting to remove the line above. > Thanks for the commit pointer. > > No, the BPF program does not depend on fwmark_reflect. It depends > on the sk->sk_mark set by a user space process. Then I don't fully understand, as ip_send_unicast_reply is only called with the per-netns percpu ctl_sk. > I was also considering to do ipcm_init_sk() but then rolled back > because of the global control sock here. > > > > > ipv6 seems to work differently enough not to have this problem, > > tcp_v6_send_response passing fl6.flowi6_mark directly to ip6_xmit. > > This was added in commit commit 92e55f412cff ("tcp: don't annotate > > mark on control socket from tcp_v6_send_response()"). > Correct. IPv6 does it differently, so the same problem is > not observed in IPv6. > > > > > But I do see the same pattern where a socket mark is set from a > > reflected value in icmp_reply and __icmp_send. Those almost certainly > > need updating too. I can do that separately if you prefer. I even > > placed ipcm_init right below this sk_mark initialization without > > considering ipcm_init_sk. D'oh. > Good point. I think it will only be a few lines change altogether, > so it makes little sense to break up the fix. I will toss mine and > wait for yours ;) Will do. Want to double check my initial rushed reading first. > Thanks for your help! Not at all. Apologies for the breakage.. Thanks for the initial fix! ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet 2020-07-01 0:45 ` Willem de Bruijn @ 2020-07-01 1:22 ` Martin KaFai Lau 0 siblings, 0 replies; 5+ messages in thread From: Martin KaFai Lau @ 2020-07-01 1:22 UTC (permalink / raw) To: Willem de Bruijn; +Cc: Network Development, David Miller, Kernel Team On Tue, Jun 30, 2020 at 08:45:13PM -0400, Willem de Bruijn wrote: > On Tue, Jun 30, 2020 at 7:46 PM Martin KaFai Lau <kafai@fb.com> wrote: > > > > On Tue, Jun 30, 2020 at 07:20:46PM -0400, Willem de Bruijn wrote: > > > On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@fb.com> wrote: > > > > > > > > When testing a recent kernel (5.6 in our case), the skb->mark of the > > > > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark. It is > > > > discovered by the bpf@tc that depends on skb->mark to work properly. > > > > The same bpf prog has been working in the earlier kernel version. > > > > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"), > > > > the skb->mark is set and seen by bpf@tc properly. > > > > > > > > We have noticed that in IPv4 TCP RST but it should also > > > > happen to the ACK based on tcp_v4_send_ack() is also depending > > > > on ip_send_unicast_reply(). > > > > > > > > This patch tries to fix it by initializing the ipc.sockc.mark to > > > > fl4.flowi4_mark. > > > > > > > > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg") > > > > Cc: Willem de Bruijn <willemb@google.com> > > > > Signed-off-by: Martin KaFai Lau <kafai@fb.com> > > > > --- > > > > net/ipv4/ip_output.c | 1 + > > > > 1 file changed, 1 insertion(+) > > > > > > > > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c > > > > index 090d3097ee15..033512f719ec 100644 > > > > --- a/net/ipv4/ip_output.c > > > > +++ b/net/ipv4/ip_output.c > > > > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb, > > > > sk->sk_bound_dev_if = arg->bound_dev_if; > > > > sk->sk_sndbuf = sysctl_wmem_default; > > > > sk->sk_mark = fl4.flowi4_mark; > > > > + ipc.sockc.mark = fl4.flowi4_mark; > > > > err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base, > > > > len, 0, &ipc, &rt, MSG_DONTWAIT); > > > > if (unlikely(err)) { > > > > > > Yes, this total sense. I missed these cases. > > > > > > Slight modification, the line above then no longer needs to be set. > > > That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation > > > with fwmark_reflect enabled"). Basically, it pretends that the socket > > > has a mark associated, but sk here is always the (netns) global > > > control sock. So your BPF program was depending on fwmark_reflect? > > Make sense. I was also tempting to remove the line above. > > Thanks for the commit pointer. > > > > No, the BPF program does not depend on fwmark_reflect. It depends > > on the sk->sk_mark set by a user space process. > > Then I don't fully understand, as ip_send_unicast_reply is only called > with the per-netns percpu ctl_sk. Before this proposed patch, my understanding is, the ctl_sk->sk_mark is correctly set here in ip_send_unicast_reply(). The ctl_sk->sk_mark was actually set earlier in the tcp_v4_send_reset(). However, ctl_sk->sk_mark is not used to set the skb->mark. Instead, cork->mark is now used to initialize the skb->mark in __ipmake_skb(). The cork->mark is not properly set in ip_setup_cork() because ipc->sockc.mark is 0 here which this proposed patch is trying to address. The call stack is something like this: tcp_v4_send_reset() ctl_sk->sk_mark = (sk->sk_state == TCP_TIME_WAIT) ?... /* <- sk_mark is set */ => ip_send_unicast_reply() => ip_append_data(..., &ipc, ...); => ip_setup_cork(..., &inet->cork.base, ipc, ...); cork->mark = ipc->sockc.mark; /* <-- ipc->sockc.mark is 0 */ => ip_push_pending_frames(sk, &fl4); => ip_finish_skb(sk, fl4) => __ip_make_skb(sk, fl4, ..., &inet_sk(sk)->cork.base); skb->mark = cork->mark; /* <-- cork->mark is 0 */ ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-07-01 1:22 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-06-30 22:18 [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet Martin KaFai Lau 2020-06-30 23:20 ` Willem de Bruijn 2020-06-30 23:45 ` Martin KaFai Lau 2020-07-01 0:45 ` Willem de Bruijn 2020-07-01 1:22 ` Martin KaFai Lau
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.