From: wenxu <wenxu@ucloud.cn>
To: Paul Blakey <paulb@mellanox.com>
Cc: "pablo@netfilter.org" <pablo@netfilter.org>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
Mark Bloch <markb@mellanox.com>
Subject: Re: Question about flow table offload in mlx5e
Date: Wed, 27 Nov 2019 20:16:38 +0800 [thread overview]
Message-ID: <dc72770c-8bc3-d302-be73-f19f9bbe269f@ucloud.cn> (raw)
In-Reply-To: <84874b42-c525-2149-539d-e7510d15f6a6@mellanox.com>
Sorry maybe something mess you, Ignore with my patches.
I also did the test like you with route tc rules to ft callback.
please also did the following test: mlx_p0 is the pf and mlx_pf0vf0 is the vf .
ifconfig mlx_p0 172.168.152.75/24 up
ip n replace 172.168.152.241 dev mlx_p0 lladdr aa:bb:cc:dd:ee:ff
ip l add dev tun1 type gretap external
tc qdisc add dev tun1 ingress
tc qdisc add dev mlx_pf0vf0 ingress
tc filter add dev mlx_pf0vf0 pref 2 ingress protocol ip flower skip_sw
ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
tcp_flags 0/0x5 action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
nocsum pipe action mirred egress redirect dev tun1
In the virtual machine:
ifconfig eth0 10.0.0.75/24
ip r a default via 10.0.0.1
ip n replace 10.0.0.1 dev eth0 lladdr aa:bb:cc:dd:ee:01
iperf -c 10.0.1.241 -i 2 -B 10.0.0.75:5002 -t 10
The script above can offload the syn packets, The packet can't be captured on mlx_pf0vf0.
I think the rule is ok. The problem is that if I add another rule in device tun1 as following
It will lead the syn packet can't be offloaded
tc filter add dev tun1 pref 2 ingress protocol ip flower ip_proto tcp src_ip
10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
action mirred egress redirect dev mlx_pf0vf0
在 2019/11/27 19:51, Paul Blakey 写道:
> Sorry I didn't have time apply your patches.
>
> I did test it again with route tc rules to ft callback, here's the diff:
>
> @@ -1291,7 +1304,7 @@ static int mlx5e_rep_setup_tc(struct net_device *dev, enum tc_setup_type type,
> case TC_SETUP_BLOCK:
> return flow_block_cb_setup_simple(type_data,
> &mlx5e_rep_block_tc_cb_list,
> - mlx5e_rep_setup_tc_cb,
> + mlx5e_rep_setup_ft_cb,
> priv, priv, true);
> case TC_SETUP_FT:
> return flow_block_cb_setup_simple(type_data,
>
>
> I ran this script after creating a VF (ens1f2) and entering switchdev mode (creating represntor ens1f0_0):
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> ifconfig tun1 up
>
> ifconfig ens1f0_0 0 up
> tc qdisc add dev ens1f0_0 ingress
>
> ifconfig ens1f0 172.168.152.75/24 up
> ip n replace 172.168.162.241 dev ens1f0 lladdr aa:bb:cc:dd:ee:01
>
> tc filter del dev ens1f0_0 ingress
>
> tc filter add dev ens1f0_0 pref 2 ingress proto ip flower \
> skip_sw \
> ip_proto tcp dst_ip 5.5.5.6 src_ip 5.5.5.5 tcp_flags 0/0x5 \
> action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000 nocsum pipe \
> action mirred egress redirect dev tun1
>
> tc filter add dev ens1f0_0 pref 1 ingress proto ip flower \
> skip_hw \
> action drop
>
> ifconfig ens1f2 5.5.5.5/24 up
> ip n replace 5.5.5.6 dev ens1f2 lladdr aa:bb:cc:dd:ee:ff
>
> timeout 3 iperf -c 5.5.5.6
>
> tc -s filter show dev ens1f0_0 ingress
>
>
>
> Notice I run iperf client without Iperf server, so I get only syn packets.
>
> Here is the tcpdump on the VF (ens1f2):
>
> Executing: sudo tcpdump -nnep -i ens1f2
>
> Executing: sudo tcpdump -nnep -i ens1f2
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on ens1f2, link-type EN10MB (Ethernet), capture size 262144 bytes
> 13:49:10.610376 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249857484 ecr 0,nop,wscale 7], length 0
> 13:49:11.616262 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249858489 ecr 0,nop,wscale 7], length 0
> 13:49:13.664261 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249860537 ecr 0,nop,wscale 7], length 0
> 13:49:17.696261 24:8a:07:a5:28:01 > aa:bb:cc:dd:e
>
> I get:
>
> filter protocol ip pref 1 flower chain 0
> filter protocol ip pref 1 flower chain 0 handle 0x1
> eth_type ipv4
> skip_hw
> not_in_hw
> action order 1: gact action drop
> random type none pass val 0
> index 1 ref 1 bind 1 installed 3 sec used 3 sec
> Action statistics:
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
> eth_type ipv4
> ip_proto tcp
> dst_ip 5.5.5.6
> src_ip 5.5.5.5
> tcp_flags 0/5
> skip_sw
> in_hw
> action order 1: tunnel_key set
> src_ip 0.0.0.0
> dst_ip 172.168.152.241
> key_id 1000
> nocsum pipe
> index 1 ref 1 bind 1 installed 3 sec used 3 sec
> Action statistics:
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>
> action order 2: mirred (Egress Redirect to device tun1) stolen
> index 1 ref 1 bind 1 installed 3 sec used 1 sec
> Action statistics:
> Sent 232 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>
>
> And it counts the 2 syn packets in hardware, the packets are leaving the VF (ens1f2) and not arriving at the mlx5 representor device (ens1f0_0),
>
> which means hardware got them. It's also couned in the above encap rule. And the software only (skip_hw, prio 1) rule didn't catch any packets.
>
>
> Thanks,
>
> Paul.
>
>
>
> On 11/26/2019 10:18 AM, wenxu wrote:
>
> Hi Paul,
>
> Did your test for this case? There are some problem that I reported?
>
>
> BR
>
> wenxu
>
> On 11/24/2019 4:46 PM, Paul Blakey wrote:
>
>
> Hi,
>
> The syn packet might not be actually offloaded because there isn't a neighbor to resolve the destination mac for the tunnel destination ip (next hop mac).
> Try setting the neighbor via "ip neigh replace dev mlx5_p0 172.168.152.241 lladdr <next hop mac>"
> Or running ping to 172.168.152.241 before adding (or in background) the rule to resolve the mac and make available.
> I'll test it on my end.
>
>
> Thanks,
> Paul.
>
>
>
>
> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> Sent: Friday, November 22, 2019 8:26 AM
> To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>; Mark Bloch
> <markb@mellanox.com><mailto:markb@mellanox.com>
> Subject: Re: Question about flow table offload in mlx5e
>
> Hi Paul,
>
>
> There are some update. I also test it through replacing mlx5e_rep_setup_tc
> _cb with mlx5e_rep_setup_ft_cb
>
>
> ifconfig mlx_p0 172.168.152.75/24 up
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
>
> tc filter add dev mlx_pf0vf0 pref 2 ingress protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5 action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1
>
> tc filter add dev tun1 pref 2 ingress protocol ip flower ip_proto tcp src_ip
> 10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
> enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
> action mirred egress redirect dev mlx_pf0vf0
>
>
> If you run this script on the host， and in the virtual machine run "iperf -c
> 10.0.1.241 -i 2 -B 10.0.0.75:5002 -t 1000"
>
> The tcp syn packet will not be offloaded
>
>
> But if you only run the script without the last filter as following , The tcp syn
> packet will be offloaded.
>
> ifconfig mlx_p0 172.168.152.75/24 up
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
>
> tc filter add dev mlx_pf0vf0 pref 2 ingress protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5 action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1.
>
> I think there are some problem in mlx5e_rep_setup_ft_cb.
>
> On 11/21/2019 9:05 PM, Paul Blakey wrote:
>
>
> I see, I will test that, and how about normal FWD rules?
>
> Paul.
>
>
>
> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> Sent: Thursday, November 21, 2019 2:35 PM
> To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
> Cc: pablo@netfilter.org<mailto:pablo@netfilter.org> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ;
> netdev@vger.kernel.org<mailto:netdev@vger.kernel.org> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
> <markb@mellanox.com><mailto:markb@mellanox.com> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
> Subject: Re: Question about flow table offload in mlx5e
>
>
> 在 2019/11/21 19:39, Paul Blakey 写道:
>
> They are good fixes, exactly what we had when we
> tested this, thanks.
>
> Regarding encap, I don't know what changes you did,
> how does the encap
>
> rule look? Is it a FWD to vxlan device? If not it should be, as
> our driver
> expects that.
> It is fwd to a gretap devices
>
>
> I tried it on my setup via tc, by changing the callback
> of tc
>
> (mlx5e_rep_setup_tc_cb) to that of ft
> (mlx5e_rep_setup_ft_cb),
>
> and testing a vxlan encap rule:
> sudo tc qdisc add dev ens1f0_0 ingress
> sudo ifconfig ens1f0 7.7.7.7/24 up
> sudo ip link add name vxlan0 type vxlan dev ens1f0
> remote 7.7.7.8 dstport
>
> 4789 external
>
> sudo ifconfig vxlan0 up
> sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0
> protocol ip flower
>
> dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw action
> tunnel_key set
> src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action
> mirred egress
> redirect dev vxlan
>
>
> then tc show:
> filter protocol ip pref 1 flower chain 0 handle 0x1
> dst_mac aa:bb:cc:dd:ee:ff
>
> ip_proto udp skip_sw in_hw in_hw_count 1
>
> tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id
> 1234 dst_port 4789
>
> csum pipe
>
> Stats: used 119 sec 0 pkt
> mirred (Egress Redirect to device vxlan0)
> Stats: used 119 sec 0 pkt
>
>
> Can you send packet that match this offloaded flow to check
> it is real
> offloaded?
>
> In the flowtable offload with my patches both
> TC_SETUP_BLOCK and
> TC_SETUP_FT can offload the rule success
>
> But in the TC_SETUP_FT case the packet is not real offloaded.
>
>
> I will test like u did.
>
>
>
>
>
>
> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> Sent: Thursday, November 21, 2019 10:29 AM
> To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
> Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>
> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>
> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
> <markb@mellanox.com><mailto:markb@mellanox.com>
> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
> Subject: Re: Question about flow table
> offload in mlx5e
>
>
> On 11/21/2019 3:42 PM, Paul Blakey wrote:
>
> Hi,
>
> The original design was the block
> setup to use TC_SETUP_FT type, and
>
> the
>
> tc event type to be case
> TC_SETUP_CLSFLOWER.
>
> We will post a patch to change that. I
> would advise to wait till we fix that
>
> 😊
>
> I'm not sure how you get to this
> function mlx5e_rep_setup_ft_cb() if it
>
> the
>
> nf_flow_table_offload ndo_setup_tc event
> was TC_SETUP_BLOCK, and
>
> not
>
> TC_SETUP_FT.
>
>
> Yes I change the TC_SETUP_BLOCK to
> TC_SETUP_FT in the
> nf_flow_table_offload_setup.
>
> Two fixes patch provide:
>
> http://patchwork.ozlabs.org/patch/1197818/
>
> http://patchwork.ozlabs.org/patch/1197876/
>
> So this change made by me is not correct
> currently?
>
>
> In our driver en_rep.c we have:
>
> -------switch (type) {
> -------case TC_SETUP_BLOCK:
> ------->-------return
> flow_block_cb_setup_simple(type_data,
> ------->------->------->------->------->---
> ----
>
> &mlx5e_rep_block_tc_cb_list,
>
> ------->------->------->------->------->---
> ---- mlx5e_rep_setup_tc_cb,
> ------->------->------->------->------->---
> ---- priv, priv, true);
> -------case TC_SETUP_FT:
> ------->-------return
> flow_block_cb_setup_simple(type_data,
> ------->------->------->------->------->---
> ----
>
> &mlx5e_rep_block_ft_cb_list,
>
> ------->------->------->------->------->---
> ---- mlx5e_rep_setup_ft_cb,
> ------->------->------->------->------->---
> ---- priv, priv, true);
> -------default:
> ------->-------return -EOPNOTSUPP;
> -------}
>
> In nf_flow_table_offload.c:
>
> -------bo.binder_type>-=
>
> FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>
> -------bo.extack>------= &extack;
> -------INIT_LIST_HEAD(&bo.cb_list);
> -------err = dev->netdev_ops-
>
>
> ndo_setup_tc(dev, TC_SETUP_BLOCK,
>
>
> &bo);
>
> -------if (err < 0)
> ------->-------return err;
> -------return
> nf_flow_table_block_setup(flowtable, &bo, cmd);
>
> }
>
> EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>
>
> So unless you changed that as well,
> you should have gotten to
>
> mlx5e_rep_setup_tc_cb and not
> mlx5e_rep_setup_tc_ft.
>
> Regarding the encap action, there
> should be no difference on which
>
> chain
>
> the rule is on.
>
>
> But for the same encap rule can be real
> offloaded when setup through
> through TC_SETUP_BLOCK. But TC_SETUP_FT
> can't.
>
> So it is the problem of TC_SETUP_FT in
> mlx5e_rep_setup_ft_cb ?
>
>
>
>
> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> Sent: Thursday, November 21, 2019
> 9:30 AM
> To: Paul Blakey
> <paulb@mellanox.com><mailto:paulb@mellanox.com> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
> Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>
> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>
> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
> <markb@mellanox.com><mailto:markb@mellanox.com>
> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
> Subject: Question about flow table
> offload in mlx5e
>
> Hi paul,
>
> The flow table offload in the mlx5e is
> based on TC_SETUP_FT.
>
>
> It is almost the same as
> TC_SETUP_BLOCK.
>
> It just set
> MLX5_TC_FLAG(FT_OFFLOAD) flags and change
> cls_flower.common.chain_index =
> FDB_FT_CHAIN;
>
> In following codes line 1380 and 1392
>
> 1368 static int
> mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
> *type_data,
> 1369 void *cb_priv)
> 1370 {
> 1371 struct flow_cls_offload *f =
> type_data;
> 1372 struct flow_cls_offload
> cls_flower;
> 1373 struct mlx5e_priv *priv =
> cb_priv;
> 1374 struct mlx5_eswitch *esw;
> 1375 unsigned long flags;
> 1376 int err;
> 1377
> 1378 flags =
> MLX5_TC_FLAG(INGRESS) |
> 1379
> MLX5_TC_FLAG(ESW_OFFLOAD) |
> 1380
> MLX5_TC_FLAG(FT_OFFLOAD);
> 1381 esw = priv->mdev-
>
>
> priv.eswitch;
>
>
> 1382
> 1383 switch (type) {
> 1384 case
> TC_SETUP_CLSFLOWER:
> 1385 if
> (!mlx5_eswitch_prios_supported(esw) || f-
>
> common.chain_index)
>
> 1386 return -
> EOPNOTSUPP;
> 1387
> 1388 /* Re-use tc offload
> path by moving the ft flow to the
> 1389 * reserved ft chain.
> 1390 */
> 1391 memcpy(&cls_flower, f,
> sizeof(*f));
> 1392
> cls_flower.common.chain_index = FDB_FT_CHAIN;
> 1393 err =
> mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>
> flags);
>
> 1394 memcpy(&f->stats,
> &cls_flower.stats, sizeof(f->stats));
>
>
> I want to add tunnel offload support
> in the flow table, I add some
>
> patches
>
> in
>
> nf_flow_table_offload.
>
> Also add the indr setup support in the
> mlx driver. And Now I can flow
>
> table
>
> offload with decap.
>
>
> But I meet a problem with the encap.
> The encap rule can be added in
> hardware successfully But it can't be
> offloaded.
>
> But I think the rule I added is correct.
> If I mask the line 1392. The rule
>
> also
>
> can
>
> be add success and can be offloaded.
>
> So there are some limit for encap
> operation for FT_OFFLOAD in
> FDB_FT_CHAIN?
>
>
> BR
>
> wenxu
>
>
next prev parent reply other threads:[~2019-11-27 12:17 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-19 7:08 [PATCH net-next] ip_gre: Make none-tun-dst gre tunnel keep tunnel info wenxu
2019-11-20 0:39 ` David Miller
2019-11-21 7:30 ` Question about flow table offload in mlx5e wenxu
2019-11-21 7:42 ` Paul Blakey
2019-11-21 8:28 ` wenxu
2019-11-21 11:39 ` Paul Blakey
2019-11-21 11:40 ` Paul Blakey
2019-11-21 12:35 ` wenxu
2019-11-21 13:05 ` Paul Blakey
2019-11-21 13:39 ` wenxu
2019-11-22 6:12 ` wenxu
[not found] ` <64285654-bc9a-c76e-5875-dc6e434dc4d4@ucloud.cn>
2019-11-24 8:46 ` Paul Blakey
2019-11-24 11:14 ` wenxu
2019-11-26 8:18 ` wenxu
[not found] ` <84874b42-c525-2149-539d-e7510d15f6a6@mellanox.com>
2019-11-27 12:16 ` wenxu
2019-11-27 12:16 ` wenxu [this message]
2019-11-27 13:11 ` Paul Blakey
2019-11-27 13:20 ` Paul Blakey
2019-11-27 13:45 ` wenxu
2019-12-02 3:37 ` wenxu
2019-12-05 15:17 ` Paul Blakey
2019-12-08 9:39 ` Paul Blakey
2019-12-09 3:18 ` wenxu
2019-12-09 7:48 ` Paul Blakey
2019-12-09 10:48 ` wenxu
2019-12-10 6:53 ` wenxu
2019-11-28 5:03 ` Bad performance for VF outgoing in offloaded mode wenxu
2019-12-04 13:50 ` Roi Dayan
2019-12-04 14:32 ` wenxu
2019-12-05 3:41 ` wenxu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dc72770c-8bc3-d302-be73-f19f9bbe269f@ucloud.cn \
--to=wenxu@ucloud.cn \
--cc=markb@mellanox.com \
--cc=netdev@vger.kernel.org \
--cc=pablo@netfilter.org \
--cc=paulb@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).