How to read from pkt_end?

How to read from pkt_end?

[Srivats P]

Hi,

How do I read from the end of the packet in a XDP program? I tried the
below ebpf program to read the last 4 bytes of the packet, but the
verifier rejects it.

Program
=======
__section("prog")
int xdp_prog(struct xdp_md *ctx)
{
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    __u16 len = ((long)ctx->data_end - (long)ctx->data);
    __u16 ofs = len - 4;
    __u32 *mgc = (__u32*)(data+ofs);

    if ((data + len) > data_end)
        return XDP_ABORTED;

    if (*mgc == 0x1d10c0da)
        return XDP_DROP;

    return XDP_PASS;
}

llvm-objdump generated disassembly
==================================
Disassembly of section prog:
xdp_prog:
; {
       0:       r0 = 0
; void *data = (void *)(long)ctx->data;
       1:       r2 = *(u32 *)(r1 + 0)
; void *data_end = (void *)(long)ctx->data_end;
       2:       r3 = *(u32 *)(r1 + 4)
; __u16 len = ((long)ctx->data_end - (long)ctx->data);
       3:       r1 = r3
       4:       r1 -= r2
; __u16 ofs = len - 4;
       5:       r4 = r1
       6:       r4 &= 65535
; if ((data + len) > data_end)
       7:       r5 = r2
       8:       r5 += r4
       9:       if r5 > r3 goto +7 <LBB0_3>
; __u16 ofs = len - 4;
      10:       r1 += 65532
; __u32 *mgc = (__u32*)(data+ofs);
      11:       r1 &= 65535
      12:       r2 += r1
; if (*mgc == 0x1d10c0da)
      13:       r1 = *(u32 *)(r2 + 0)
      14:       r0 = 1
; return XDP_DROP;
      15:       if r1 == 487637210 goto +1 <LBB0_3>
      16:       r0 = 2

LBB0_3:
; }
      17:       exit


Verifier output (via ip link)
=============================
Prog section 'prog' rejected: Permission denied (13)!
 - Type:         6
 - Instructions: 18 (0 over limit)
 - License:

Verifier analysis:

0: (b7) r0 = 0
1: (61) r2 = *(u32 *)(r1 +0)
2: (61) r3 = *(u32 *)(r1 +4)
3: (bf) r1 = r3
4: (1f) r1 -= r2
5: (bf) r4 = r1
6: (57) r4 &= 65535
7: (bf) r5 = r2
8: (0f) r5 += r4
last_idx 8 first_idx 0
regs=10 stack=0 before 7: (bf) r5 = r2
regs=10 stack=0 before 6: (57) r4 &= 65535
regs=10 stack=0 before 5: (bf) r4 = r1
regs=2 stack=0 before 4: (1f) r1 -= r2
regs=6 stack=0 before 3: (bf) r1 = r3
regs=c stack=0 before 2: (61) r3 = *(u32 *)(r1 +4)
regs=4 stack=0 before 1: (61) r2 = *(u32 *)(r1 +0)
9: (2d) if r5 > r3 goto pc+7
 R0_w=inv0 R1_w=inv(id=0) R2_w=pkt(id=0,off=0,r=0,imm=0)
R3_w=pkt_end(id=0,off=0,imm=0)
R4_w=invP(id=0,umax_value=65535,var_off=(0x0; 0xffff))
R5_w=pkt(id=1,off=0,r=0,umax_value=65535,var_off=(0x0; 0xffff))
R10=fp0
10: (07) r1 += 65532
11: (57) r1 &= 65535
12: (0f) r2 += r1
last_idx 12 first_idx 0
regs=2 stack=0 before 11: (57) r1 &= 65535
regs=2 stack=0 before 10: (07) r1 += 65532
regs=2 stack=0 before 9: (2d) if r5 > r3 goto pc+7
regs=2 stack=0 before 8: (0f) r5 += r4
regs=2 stack=0 before 7: (bf) r5 = r2
regs=2 stack=0 before 6: (57) r4 &= 65535
regs=2 stack=0 before 5: (bf) r4 = r1
regs=2 stack=0 before 4: (1f) r1 -= r2
regs=6 stack=0 before 3: (bf) r1 = r3
regs=c stack=0 before 2: (61) r3 = *(u32 *)(r1 +4)
regs=4 stack=0 before 1: (61) r2 = *(u32 *)(r1 +0)
13: (61) r1 = *(u32 *)(r2 +0)
invalid access to packet, off=0 size=4, R2(id=2,off=0,r=0)
R2 offset is outside of the packet
processed 14 insns (limit 1000000) max_states_per_insn 0 total_states
0 peak_states 0 mark_read 0

Error fetching program/map!

I did read the filter.txt documentation, but couldn't find anything
pertinent to this case where instead of adding fixed (literal value)
offsets to pkt, we want to work backwards from pkt_end - the above
program will always read within packet boundaries, but the length of
the packet can of course vary with each packet.

I also searched the list archives but couldn't find anything related.

I'm using Kernel version 5.3 (stock kernel with Ubuntu 18.04 LTS)

Srivats

Re: How to read from pkt_end?

[Srivats P]

Hi,

> How do I read from the end of the packet in a XDP program? I tried the
> below ebpf program to read the last 4 bytes of the packet, but the
> verifier rejects it.

Got it working with some trial and error - the trick was to AND the
calculated offset with 0x3fff - largest value for a packet size that
allows 9K jumbo.

Here's the working code -

__section("prog")
int xdp_prog(struct xdp_md *ctx)
{
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    __u16 len = data_end - data;
    if ((data + len) > data_end)
        return XDP_ABORTED;

    __u16 ofs = (len - 4) & 0x3fff;
    __u32 *mgc = (__u32*)(data+ofs);
    if ((mgc + 1) > (__u32*)data_end)
        return XDP_ABORTED;

    if (*mgc == 0x1d10c0da)
        return XDP_DROP;

    return XDP_PASS;
}

What clued me onto the AND thing was the following lines in
Documentation/networking/filter.txt about overflow -

    Operation 'r3 += rX' may overflow and become less than original skb->data,
    therefore the verifier has to prevent that.  So when it sees 'r3 += rX'
    instruction and rX is more than 16-bit value, any subsequent
bounds-check of r3
    against skb->data_end will not give us 'range' information, so
attempts to read
    through the pointer will give "invalid access to packet" error.

Without the (superfluous) data + len > data_end check, LLVM was
optimizing the code in a way that led to scalar arithmetic which the
verifier wouldn't allow.

Of course, the ideal way to work backwards from the end of  the packet
is to allow arithmetic on pkt_end (so you could do `ofs = data_end -
4`), but there may be valid reasons not to allow that, that I'm not
aware of.

Srivats