From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Daniel Borkmann <daniel@iogearbox.net>
Cc: "Lorenz Bauer" <lmb@cloudflare.com>,
"Maciej Żenczykowski" <maze@google.com>,
"Saeed Mahameed" <saeed@kernel.org>,
"Daniel Borkmann" <borkmann@iogearbox.net>,
"Alexei Starovoitov" <alexei.starovoitov@gmail.com>,
BPF-dev-list <bpf@vger.kernel.org>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
"Lorenzo Bianconi" <lorenzo.bianconi@redhat.com>,
"John Fastabend" <john.fastabend@gmail.com>,
"Jakub Kicinski" <kuba@kernel.org>,
"Shaun Crampton" <shaun@tigera.io>,
"David Miller" <davem@davemloft.net>,
"Marek Majkowski" <marek@cloudflare.com>,
brouer@redhat.com
Subject: Re: BPF redirect API design issue for BPF-prog MTU feedback?
Date: Mon, 21 Sep 2020 18:26:38 +0200 [thread overview]
Message-ID: <20200921182638.5d8343fd@carbon> (raw)
In-Reply-To: <340f209d-58d4-52a6-0804-7102d80c1468@iogearbox.net>
On Mon, 21 Sep 2020 17:08:17 +0200
Daniel Borkmann <daniel@iogearbox.net> wrote:
> On 9/21/20 2:49 PM, Jesper Dangaard Brouer wrote:
> > On Mon, 21 Sep 2020 11:37:18 +0100
> > Lorenz Bauer <lmb@cloudflare.com> wrote:
> >> On Sat, 19 Sep 2020 at 00:06, Maciej Żenczykowski <maze@google.com> wrote:
> >>>
> >>>> This is a good point. As bpf_skb_adjust_room() can just be run after
> >>>> bpf_redirect() call, then a MTU check in bpf_redirect() actually
> >>>> doesn't make much sense. As clever/bad BPF program can then avoid the
> >>>> MTU check anyhow. This basically means that we have to do the MTU
> >>>> check (again) on kernel side anyhow to catch such clever/bad BPF
> >>>> programs. (And I don't like wasting cycles on doing the same check two
> >>>> times).
> >>>
> >>> If you get rid of the check in bpf_redirect() you might as well get
> >>> rid of *all* the checks for excessive mtu in all the helpers that
> >>> adjust packet size one way or another way. They *all* then become
> >>> useless overhead.
> >>>
> >>> I don't like that. There may be something the bpf program could do to
> >>> react to the error condition (for example in my case, not modify
> >>> things and just let the core stack deal with things - which will
> >>> probably just generate packet too big icmp error).
> >>>
> >>> btw. right now our forwarding programs first adjust the packet size
> >>> then call bpf_redirect() and almost immediately return what it
> >>> returned.
> >>>
> >>> but this could I think easily be changed to reverse the ordering, so
> >>> we wouldn't increase packet size before the core stack was informed we
> >>> would be forwarding via a different interface.
> >>
> >> We do the same, except that we also use XDP_TX when appropriate. This
> >> complicates the matter, because there is no helper call we could
> >> return an error from.
> >
> > Do notice that my MTU work is focused on TC-BPF. For XDP-redirect the
> > MTU check is done in xdp_ok_fwd_dev() via __xdp_enqueue(), which also
> > happens too late to give BPF-prog knowledge/feedback. For XDP_TX I
> > audited the drivers when I implemented xdp_buff.frame_sz, and they
> > handled (or I added) handling against max HW MTU. E.g. mlx5 [1].
> >
> > [1] https://elixir.bootlin.com/linux/v5.9-rc6/source/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#L267
> >
> >> My preference would be to have three helpers: get MTU for a device,
> >> redirect ctx to a device (with MTU check), resize ctx (without MTU
> >> check) but that doesn't work with XDP_TX. Your idea of doing checks
> >> in redirect and adjust_room is pragmatic and seems easier to
> >> implement.
> >
> > I do like this plan/proposal (with 3 helpers), but it is not possible
> > with current API. The main problem is the current bpf_redirect API
> > doesn't provide the ctx, so we cannot do the check in the BPF-helper.
> >
> > Are you saying we should create a new bpf_redirect API (that incl packet ctx)?
>
> Sorry for jumping in late here... one thing that is not clear to me
> is that if we are fully sure that skb is dropped by stack anyway due
> to invalid MTU (redirect to ingress does this via dev_forward_skb(),
Yes, TC-redirecting to *INGRESS* have a slightly relaxed MTU check via
is_skb_forwardable() called via ____dev_forward_skb(). This MTU check
seems redundant as netstack will do MTU checks anyhow.
> it's not fully clear to me whether it's also the case for the
> dev_queue_xmit()),
This seems the problematic case; TC-ingress redirect to netdev-egress,
that basically calls dev_queue_xmit(). I tried to follow the code all
the way into ixgbe driver, and I didn't see any MTU checks.
We might have to add a MTU check here, as it could be considered a
bug/problematic that we allow this. (e.g. netdev with large MTU can
redirect frames larger than MTU of egress netdev).
> then why not dropping all the MTU checks aside
> from SKB_MAX_ALLOC sanity check for BPF helpers
I agree, and think that the MTU checks in the BPF-helpers, make little
sense, as we have found ways to circumvent these checks (as discussed
in thread).
> and have something like a device object (similar to e.g. TCP sockets)
> exposed to BPF prog where we can retrieve the object and read
> dev->mtu from the prog, so the BPF program could then do the
> "exception" handling internally w/o extra prog needed (we also
> already expose whether skb is GSO or not).
I do think we need some BPF-helper that allows BPF-prog to lookup MTU
of a netdev, so it can do proper ICMP exception handling.
I looked at doing ICMP exception handling on kernel-side, but realized
that this is not possible at the TC-redirect layer, as we have not
decoded the L3 protocol at this point (e.g. cannot know if I need to
call icmp_send or icmp6_send).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
next prev parent reply other threads:[~2020-09-21 16:27 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-17 12:38 BPF redirect API design issue for BPF-prog MTU feedback? Jesper Dangaard Brouer
2020-09-17 12:54 ` Maciej Żenczykowski
2020-09-17 19:11 ` Saeed Mahameed
2020-09-18 10:00 ` Jesper Dangaard Brouer
2020-09-18 10:34 ` Toke Høiland-Jørgensen
2020-09-18 23:06 ` Maciej Żenczykowski
2020-09-21 10:37 ` Lorenz Bauer
2020-09-21 12:49 ` Jesper Dangaard Brouer
2020-09-21 15:08 ` Daniel Borkmann
2020-09-21 16:21 ` Marek Zavodsky
2020-09-21 21:17 ` Willem de Bruijn
2020-09-22 9:15 ` Jesper Dangaard Brouer
2020-09-21 16:26 ` Jesper Dangaard Brouer [this message]
2020-09-22 6:56 ` Eyal Birger
2020-09-21 18:04 ` John Fastabend
2020-10-06 11:45 ` Jesper Dangaard Brouer
2020-09-21 10:42 ` Lorenz Bauer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200921182638.5d8343fd@carbon \
--to=brouer@redhat.com \
--cc=alexei.starovoitov@gmail.com \
--cc=borkmann@iogearbox.net \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=lmb@cloudflare.com \
--cc=lorenzo.bianconi@redhat.com \
--cc=marek@cloudflare.com \
--cc=maze@google.com \
--cc=netdev@vger.kernel.org \
--cc=saeed@kernel.org \
--cc=shaun@tigera.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).