From: Eric Dumazet <edumazet@google.com>
To: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: renmingshuai <renmingshuai@huawei.com>,
xiyou.wangcong@gmail.com, jiri@resnulli.us, davem@davemloft.net,
vladbu@nvidia.com, netdev@vger.kernel.org, yanan@huawei.com,
liaichun@huawei.com, caowangbao@huawei.com,
Eric Dumazet <eric.dumazet@gmail.com>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Victor Nogueira <victor@mojatatu.com>,
Pedro Tammela <pctammela@mojatatu.com>,
Davide Caratti <dcaratti@redhat.com>
Subject: Re: [PATCH] net/sched: Forbid assigning mirred action to a filter attached to the egress
Date: Wed, 20 Mar 2024 17:57:51 +0100 [thread overview]
Message-ID: <CANn89iLjK3vf-yHvKdY=wvOdEeWubB0jt2=5d-1m7dkTYBwBOg@mail.gmail.com> (raw)
In-Reply-To: <CAM0EoM=WCLvjCxkDGSEP-+NqEd2HnieiW8emNoV1LeV6n6w9VQ@mail.gmail.com>
On Tue, Mar 19, 2024 at 9:54 PM Jamal Hadi Salim <jhs@mojatatu.com> wrote:
>
> On Tue, Mar 19, 2024 at 5:38 AM Eric Dumazet <edumazet@google.com> wrote:
> >
> > On Mon, Mar 18, 2024 at 11:05 PM Jamal Hadi Salim <jhs@mojatatu.com> wrote:
> > >
> > > On Mon, Mar 18, 2024 at 3:11 PM Eric Dumazet <edumazet@google.com> wrote:
> > > >
> > > > On Mon, Mar 18, 2024 at 6:36 PM Jamal Hadi Salim <jhs@mojatatu.com> wrote:
> > > > >
> > > > > On Mon, Mar 18, 2024 at 11:46 AM Eric Dumazet <edumazet@google.com> wrote:
> > > > > >
> > > > > > On Mon, Mar 18, 2024 at 3:27 PM Jamal Hadi Salim <jhs@mojatatu.com> wrote:
> > > > > > >
> > > > > > > On Sun, Mar 17, 2024 at 12:10 PM Jamal Hadi Salim <jhs@mojatatu.com> wrote:
> > > > > > > >
> > > > > > > > On Thu, Mar 14, 2024 at 1:14 PM Jamal Hadi Salim <jhs@mojatatu.com> wrote:
> > > > > > > > >
> > > > > > > > > On Thu, Mar 14, 2024 at 7:18 AM renmingshuai <renmingshuai@huawei.com> wrote:
> > > > > > > > > >
> > > > > > > > > > As we all know the mirred action is used to mirroring or redirecting the
> > > > > > > > > > packet it receives. Howerver, add mirred action to a filter attached to
> > > > > > > > > > a egress qdisc might cause a deadlock. To reproduce the problem, perform
> > > > > > > > > > the following steps:
> > > > > > > > > > (1)tc qdisc add dev eth0 root handle 1: htb default 30 \n
> > > > > > > > > > (2)tc filter add dev eth2 protocol ip prio 2 flower verbose \
> > > > > > > > > > action police rate 100mbit burst 12m conform-exceed jump 1 \
> > > > > > > > > > / pipe mirred egress redirect dev eth2 action drop
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think you meant both to be the same device eth0 or eth2?
> > > > > > > > >
> > > > > > > > > > The stack is show as below:
> > > > > > > > > > [28848.883915] _raw_spin_lock+0x1e/0x30
> > > > > > > > > > [28848.884367] __dev_queue_xmit+0x160/0x850
> > > > > > > > > > [28848.884851] ? 0xffffffffc031906a
> > > > > > > > > > [28848.885279] tcf_mirred_act+0x3ab/0x596 [act_mirred]
> > > > > > > > > > [28848.885863] tcf_action_exec.part.0+0x88/0x130
> > > > > > > > > > [28848.886401] fl_classify+0x1ca/0x1e0 [cls_flower]
> > > > > > > > > > [28848.886970] ? dequeue_entity+0x145/0x9e0
> > > > > > > > > > [28848.887464] ? newidle_balance+0x23f/0x2f0
> > > > > > > > > > [28848.887973] ? nft_lookup_eval+0x57/0x170 [nf_tables]
> > > > > > > > > > [28848.888566] ? nft_do_chain+0xef/0x430 [nf_tables]
> > > > > > > > > > [28848.889137] ? __flush_work.isra.0+0x35/0x80
> > > > > > > > > > [28848.889657] ? nf_ct_get_tuple+0x1cf/0x210 [nf_conntrack]
> > > > > > > > > > [28848.890293] ? do_select+0x637/0x870
> > > > > > > > > > [28848.890735] tcf_classify+0x52/0xf0
> > > > > > > > > > [28848.891177] htb_classify+0x9d/0x1c0 [sch_htb]
> > > > > > > > > > [28848.891722] htb_enqueue+0x3a/0x1c0 [sch_htb]
> > > > > > > > > > [28848.892251] __dev_queue_xmit+0x2d8/0x850
> > > > > > > > > > [28848.892738] ? nf_hook_slow+0x3c/0xb0
> > > > > > > > > > [28848.893198] ip_finish_output2+0x272/0x580
> > > > > > > > > > [28848.893692] __ip_queue_xmit+0x193/0x420
> > > > > > > > > > [28848.894179] __tcp_transmit_skb+0x8cc/0x970
> > > > > > > > > >
> > > > > > > > > > In this case, the process has hold the qdisc spin lock in __dev_queue_xmit
> > > > > > > > > > before the egress packets are mirred, and it will attempt to obtain the
> > > > > > > > > > spin lock again after packets are mirred, which cause a deadlock.
> > > > > > > > > >
> > > > > > > > > > Fix the issue by forbidding assigning mirred action to a filter attached
> > > > > > > > > > to the egress.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Mingshuai Ren <renmingshuai@huawei.com>
> > > > > > > > > > ---
> > > > > > > > > > net/sched/act_mirred.c | 4 +++
> > > > > > > > > > .../tc-testing/tc-tests/actions/mirred.json | 32 +++++++++++++++++++
> > > > > > > > > > 2 files changed, 36 insertions(+)
> > > > > > > > > >
> > > > > > > > > > diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
> > > > > > > > > > index 5b3814365924..fc96705285fb 100644
> > > > > > > > > > --- a/net/sched/act_mirred.c
> > > > > > > > > > +++ b/net/sched/act_mirred.c
> > > > > > > > > > @@ -120,6 +120,10 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
> > > > > > > > > > NL_SET_ERR_MSG_MOD(extack, "Mirred requires attributes to be passed");
> > > > > > > > > > return -EINVAL;
> > > > > > > > > > }
> > > > > > > > > > + if (tp->chain->block->q->parent != TC_H_INGRESS) {
> > > > > > > > > > + NL_SET_ERR_MSG_MOD(extack, "Mirred can only be assigned to the filter attached to ingress");
> > > > > > > > > > + return -EINVAL;
> > > > > > > > > > + }
> > > > > > > > >
> > > > > > > > > Sorry, this is too restrictive as Jiri said. We'll try to reproduce. I
> > > > > > > > > am almost certain this used to work in the old days.
> > > > > > > >
> > > > > > > > Ok, i looked at old notes - it did work at "some point" pre-tdc.
> > > > > > > > Conclusion is things broke around this time frame:
> > > > > > > > https://lore.kernel.org/netdev/1431679850-31896-1-git-send-email-fw@strlen.de/
> > > > > > > > https://lore.kernel.org/netdev/1465095748.2968.45.camel@edumazet-glaptop3.roam.corp.google.com/
> > > > > > > >
> > > > > > > > Looking further into it.
> > > > > > >
> > > > > > > This is what we came up with. Eric, please take a look...
> > > > > > >
> > > > > > > cheers,
> > > > > > > jamal
> > > > > > >
> > > > > > >
> > > > > > > --- a/net/core/dev.c
> > > > > > > +++ b/net/core/dev.c
> > > > > > > @@ -3789,7 +3789,14 @@ static inline int __dev_xmit_skb(struct sk_buff
> > > > > > > *skb, struct Qdisc *q,
> > > > > > > if (unlikely(contended))
> > > > > > > spin_lock(&q->busylock);
> > > > > > >
> > > > > > > + if (dev_recursion_level()) {
> > > > > >
> > > > > > I am not sure what your intent is, but this seems wrong to me.
> > > > > >
> > > > >
> > > > > There is a deadlock if you reenter the same device which has a qdisc
> > > > > attached to it more than once.
> > > > > Essentially entering __dev_xmit_skb() we grab the root qdisc lock then
> > > > > run some action which requires it to grab the root qdisc lock (again).
> > > > > This is easy to show with mirred (although i am wondering if syzkaller
> > > > > may have produced this at some point)..
> > > > > $TC qdisc add dev $DEV root handle 1: htb default 1
> > > > > $TC filter add dev $DEV protocol ip u32 match ip protocol 1 0xff
> > > > > action mirred egress mirror dev $DEV
> > > > >
> > > > > Above example is essentially egress $DEV-> egress $DEV in both cases
> > > > > "egress $DEV" grabs the root qdisc lock. You could also create another
> > > > > example with egress($DEV1->$DEV2->back to $DEV1).
> > > > >
> > > > > > Some valid setup use :
> > > > > >
> > > > > > A bonding device, with HTB qdisc (or other qdisc)
> > > > > > (This also could be a tunnel device with a qdisc)
> > > > > >
> > > > > > -> one or multiple physical NIC, wth FQ or other qdisc.
> > > > > >
> > > > > > Packets would be dropped here when we try to reach the physical device.
> > > > > >
> > > > >
> > > > > If you have an example handy please send it. I am trying to imagine
> > > > > how those would have worked if they have to reenter the root qdisc of
> > > > > the same dev multiple times..
> > > >
> > > > Any virtual device like a GRE/SIT/IPIP/... tunnel, add a qdisc on it ?
> > > >
> > > > dev_xmit_recursion_inc() is global (per-cpu), it is not per-device.
> > > >
> > > > A stack of devices A -> B -> C would elevate the recursion level to
> > > > three just fine.
> > > >
> > > > After your patch, a stack of devices would no longer work.
> > > >
> > > > It seems mirred correctly injects packets to the top of the stack for
> > > > ingress (via netif_rx() / netif_receive_skb()),
> > > > but thinks it is okay to call dev_queue_xmit(), regardless of the context ?
> > > >
> > > > Perhaps safe-guard mirred, instead of adding more code to fast path.
> > >
> > > I agree not to penalize everybody for a "bad config" like this
> > > (surprising syzkaller hasnt caught this). But i dont see how doing the
> > > checking within mirred will catch this (we cant detect the A->B->A
> > > case).
> > > I think you are suggesting a backlog-like queue for mirred? Not far
> > > off from that is how it used to work before
> > > (https://lore.kernel.org/netdev/1465095748.2968.45.camel@edumazet-glaptop3.roam.corp.google.com/)
> >
> >
> > spin_trylock() had to go. There is no way we could keep this.
> >
>
> Not asking for it to come back... just pointing out why it worked before.
>
> > > - i.e we had a trylock for the qdisc lock and if it failed we tagged
> > > the rx softirq for a reschedule. That in itself is insufficient, we
> > > would need a loop check which is per-skb (which we had before
> > > https://lore.kernel.org/netdev/1431679850-31896-1-git-send-email-fw@strlen.de/).
> > > There are other gotchas there, potentially packet reordering.
> >
> > If we want to make sure dev_queue_xmit() is called from the top (no
> > spinlock held),
> > then we need a queue, serviced from another context.
> >
> > This extra queueing could happen if
> > __this_cpu_read(softnet_data.xmit.recursion) > 0
> >
>
> I dont see a way to detect softnet_data.xmit.recursion > 0 at mirred
> level. The first time we enter it will be 0.
Then it is fine, no qdisc spinlock is held at this point.
The second time we would
> deadlock before we hit mirred.
This is not how I see the trace.
Mirred would detect that and either drop or queue the packet to a work
queue or something.
diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
index 5b38143659249e66718348e0ec4ed3c7bc21c13d..a2c53e200629a17130f38246ab3cdb8c89c6d30e
100644
--- a/net/sched/act_mirred.c
+++ b/net/sched/act_mirred.c
@@ -237,9 +237,15 @@ tcf_mirred_forward(bool at_ingress, bool
want_ingress, struct sk_buff *skb)
{
int err;
- if (!want_ingress)
+ if (!want_ingress) {
+ if (__this_cpu_read(softnet_data.xmit.recursion) > 0) {
+ // TODO increment a drop counter perhaps ?
+ kfree_skb(skb);
+ return -EINVAL;
+ }
+
err = tcf_dev_queue_xmit(skb, dev_queue_xmit);
- else if (!at_ingress)
+ } else if (!at_ingress)
err = netif_rx(skb);
else
err = netif_receive_skb(skb);
next prev parent reply other threads:[~2024-03-20 16:58 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-14 11:17 [PATCH] net/sched: Forbid assigning mirred action to a filter attached to the egress renmingshuai
2024-03-14 11:43 ` Jiri Pirko
2024-03-14 14:04 ` renmingshuai
2024-03-14 14:47 ` Pedro Tammela
2024-03-14 17:14 ` Jamal Hadi Salim
2024-03-15 1:56 ` renmingshuai
2024-03-15 22:34 ` Jamal Hadi Salim
2024-03-17 16:10 ` Jamal Hadi Salim
2024-03-18 14:27 ` Jamal Hadi Salim
2024-03-18 15:46 ` Eric Dumazet
2024-03-18 17:36 ` Jamal Hadi Salim
2024-03-18 19:11 ` Eric Dumazet
2024-03-18 22:05 ` Jamal Hadi Salim
2024-03-19 9:38 ` Eric Dumazet
2024-03-19 20:54 ` Jamal Hadi Salim
2024-03-20 16:46 ` Jamal Hadi Salim
2024-03-20 16:57 ` Eric Dumazet [this message]
2024-03-20 17:31 ` Jamal Hadi Salim
2024-03-20 17:50 ` Jamal Hadi Salim
2024-03-20 18:13 ` Eric Dumazet
2024-03-20 18:25 ` Eric Dumazet
2024-03-20 19:34 ` Jamal Hadi Salim
2024-03-24 15:27 ` Jamal Hadi Salim
2024-03-26 23:18 ` Jamal Hadi Salim
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CANn89iLjK3vf-yHvKdY=wvOdEeWubB0jt2=5d-1m7dkTYBwBOg@mail.gmail.com' \
--to=edumazet@google.com \
--cc=caowangbao@huawei.com \
--cc=davem@davemloft.net \
--cc=dcaratti@redhat.com \
--cc=eric.dumazet@gmail.com \
--cc=jhs@mojatatu.com \
--cc=jiri@resnulli.us \
--cc=kuba@kernel.org \
--cc=liaichun@huawei.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pctammela@mojatatu.com \
--cc=renmingshuai@huawei.com \
--cc=victor@mojatatu.com \
--cc=vladbu@nvidia.com \
--cc=xiyou.wangcong@gmail.com \
--cc=yanan@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).