NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Jason A. Donenfeld]

Hello,

I am making a network device driver that receives packets in
ndo_start_xmit, "does something to them", and then sends the resultant
packet out of a kernelspace UDP socket.

The routine looks something along the lines of:

    size_t outgoing_len = calculate_outgoing_length(skb);
    struct sk_buff *outgoing = alloc_skb(outgoing_len);
    u8 *output_buffer = skb_put(outgoing, outgoing_len);

    struct scatterlist sglist[MAX_SKB_FRAGS + 1] = { 0 };
    sg_init_table(sglist, skb_shinfo(skb)->nr_frags + 1);
    skb_to_sgvec(skb, sglist, 0, skb->len);

    magic_transformer state;
    begin_magic(&state, outgoing_buffer);
    for (struct scatterlist *sg = &sglist; sg; sg = sg_next(sg)) {
        u8 *vaddr = kmap_atomic(sg_page(sg));
        update_magic(&state, vaddr + sg->offset, sg->length);
        kunmap_atomic(vaddr);
    }
    finish_magic(&state);

    send_udp(outgoing);

Hopefully that's straight-forward enough. I make the skb into an
scatterlist, and then iterate over the scatterlist, to apply a
particular transformation to each part, and then finally I send it
out.

For this, I'm using these netdev features:

    #define MY_FEATURES (NETIF_F_HW_CSUM | NETIF_F_RXCSUM | NETIF_F_SG \
                                             | NETIF_F_GSO | NETIF_F_HIGHDMA)
    dev->features |= MY_FEATURES;
    dev->hw_features |= MY_FEATURES;
    dev->hw_enc_features |= MY_FEATURES;

Using this set of features, everything works well. But the performance
isn't great. I suspect this has something to do with having to
traverse the network stack. So I've looked into offloading features.

Strangely, the performance does not change at all regardless of
whether or not NETIF_F_GSO is specified.

However, the performance becomes incredible when I use
NETIF_F_GSO_SOFTWARE instead of NETIF_F_GSO. But, when using
NETIF_F_GSO_SOFTWARE, skb->len is bigger than the MTU! This poses some
problems for me. Perhaps this is intended behavior? I'm not really
sure. My question is: how can I gain the performance benefits of
NETIF_F_GSO_SOFTWARE while still having skbs that fit inside the MTU?
And what's the difference between specifying NETIF_F_GSO and
NETIF_F_GSO_SOFTWARE?

Thank you,
Jason

Re: NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Herbert Xu]

On Wed, Nov 04, 2015 at 12:24:47PM +0100, Jason A. Donenfeld wrote:
> 
> Strangely, the performance does not change at all regardless of
> whether or not NETIF_F_GSO is specified.

The NETIF_F_GSO flag turns on software GSO which should be on
anyway.  So that could be why it seems to make no difference.

> However, the performance becomes incredible when I use
> NETIF_F_GSO_SOFTWARE instead of NETIF_F_GSO. But, when using

NETIF_F_GSO_SOFTWARE is actually a collection of bits that lists
the protocols for which we support software GSO.  The bits themselves
are in fact an indication that the hardware supports GSO directly.
So by turning them on you're electing to receive GSO packets
directly.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Jason A. Donenfeld]

Hi Herbert,

Thanks for your response!

On Thu, Nov 5, 2015 at 1:15 PM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> The NETIF_F_GSO flag turns on software GSO which should be on
> anyway.  So that could be why it seems to make no difference.

GSO is on by default? That makes sense, okay.

>
> NETIF_F_GSO_SOFTWARE is actually a collection of bits that lists
> the protocols for which we support software GSO.  The bits themselves
> are in fact an indication that the hardware supports GSO directly.
> So by turning them on you're electing to receive GSO packets
> directly.

Right -- I saw the expansion in the header file -- it gets the various
TSOs plus UFO. So what this means is that the packet hasn't yet been
split up? So were I to add this option, then my driver would have to
be responsible for splitting up the super-packets manually? In which
case, there would be no performance benefit in using it, since GSO
already does this just prior to ndo_start_xmit? Or would there be a
performance benefit in receiving the super-packets and splitting them
myself?

Re: NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Eric Dumazet]

On Thu, 2015-11-05 at 16:00 +0100, Jason A. Donenfeld wrote:

> Right -- I saw the expansion in the header file -- it gets the various
> TSOs plus UFO. So what this means is that the packet hasn't yet been
> split up? So were I to add this option, then my driver would have to
> be responsible for splitting up the super-packets manually? In which
> case, there would be no performance benefit in using it, since GSO
> already does this just prior to ndo_start_xmit? Or would there be a
> performance benefit in receiving the super-packets and splitting them
> myself?

It is a performance benefit only if you use the helpers from
net/core/tso.c as some drivers already do.

Otherwise, calling the skb_gso_segment() from your driver has no gain
compared to the one done from core networking stack.

Re: NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Jason A. Donenfeld]

On Thu, Nov 5, 2015 at 4:56 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> It is a performance benefit only if you use the helpers from
> net/core/tso.c as some drivers already do.
>
> Otherwise, calling the skb_gso_segment() from your driver has no gain
> compared to the one done from core networking stack.

Interesting, okay. It looks like it will be, in fact, useful to be
able to call skb_gso_segment() from my own driver. The reasoning is as
follows:

As mentioned, I receive packets on ndo_start_xmit, "do something to
them with function magic()", and then push them out of a UDP socket
using udp_tunnel_xmit_skb. There appears to be significant overhead
from calling udp_tunnel_xmit_skb over and over. What I'd really like
to do is pass the NETIF_F_GSO_SOFTWARE-provided super packet directly
to udp_tunnel_xmit_skb, but in fact the magic() function mentioned
above needs to work on an entire MTU-sized IP packet, not a a super
packet. So, instead, what it's looking like is:

1. Set NETIF_F_GSO_SOFTWARE to receive super packets.
2. For each super packet, break it down using skb_gso_segment().
     2a. For each segmented packet, "do my magic() function"
3. After having done the magic() to each of the segmented packets, I
*repack the results* into a new super packet. I then pass that super
packet to udp_tunnel_xmit_skb().

Is this approach a real possibility?

If so, it seems like the best way to get GSO-like qualities out of
udp_tunnel_xmit_skb. I've successfully done (1) and (2), following the
example of validate_xmit_skb() from net/core/dev.c. Now I need to
figure out how to do (3). Hopefully I'll find some nice convenience
functions for this...

Jason

Re: NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Jason A. Donenfeld]

Hi folks,

I'm still facing some considerable problems. Please see below.

On Thu, Nov 5, 2015 at 5:28 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> As mentioned, I receive packets on ndo_start_xmit, "do something to
> them with function magic()", and then push them out of a UDP socket
> using udp_tunnel_xmit_skb. There appears to be significant overhead
> from calling udp_tunnel_xmit_skb over and over. What I'd really like
> to do is pass the NETIF_F_GSO_SOFTWARE-provided super packet directly
> to udp_tunnel_xmit_skb, but in fact the magic() function mentioned
> above needs to work on an entire MTU-sized IP packet, not a a super
> packet. So, instead, what it's looking like is:
>
> 1. Set NETIF_F_GSO_SOFTWARE to receive super packets.
> 2. For each super packet, break it down using skb_gso_segment().
>      2a. For each segmented packet, "do my magic() function"
> 3. After having done the magic() to each of the segmented packets, I
> *repack the results* into a new super packet. I then pass that super
> packet to udp_tunnel_xmit_skb().
>
> Is this approach a real possibility?
>
> If so, it seems like the best way to get GSO-like qualities out of
> udp_tunnel_xmit_skb. I've successfully done (1) and (2), following the
> example of validate_xmit_skb() from net/core/dev.c. Now I need to
> figure out how to do (3). Hopefully I'll find some nice convenience
> functions for this...

So far implementing (3) is failing miserably. Is there anything wrong
with my general idea that might make this a priori impossible? For
example, will udp_tunnel_xmit_skb not accept super-packets? Or, am I
just not making use of whatever nice convenience functions are
available for constructing super-packets, and thus am doing something
wrong?

Currently, I'm doing essentially what follows below. It seems like it
ought to be working, but it's not.


    gso_segs = DIV_ROUND_UP(skb->len, skb_shinfo(skb)->gso_size);
    segs = skb_gso_segment(skb, 0);
    if (IS_ERR(segs)) {
        kfree_skb(skb);
        return PTR_ERR(segs);
    }
    dev_kfree_skb(skb);

    gso_size = magic_len(segs->len);
    len = gso_size * gso_segs;

    outgoing = alloc_skb(len, GFP_ATOMIC);
    if (!outgoing) {
        kfree_skb_list(segs);
        return -ENOMEM;
    }
    skb_shinfo(outgoing)->gso_type = SKB_GSO_UDP;
    skb_shinfo(outgoing)->gso_size = gso_size;
    skb_shinfo(outgoing)->gso_segs = 0;
    outgoing->ip_summed = CHECKSUM_PARTIAL;

    src = segs;
    while (src) {
        next = src->next;
        dst_buffer = skb_put(outgoing, magic_len(src->len));
        ret = do_magic(dst_buffer, src);
        if (ret) {
            dev_kfree_skb(outgoing);
            return ret;
        }
        ++skb_shinfo(outgoing)->gso_segs;
        src = next;
    }
    return eventually_udp_tunnel_xmit_skb(outgoing);


Let me know if there is something fundamentally wrong with this
approach. Or if you have any other pointers...

Regards,
Jason

Re: NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Herbert Xu]

On Fri, Nov 06, 2015 at 02:31:59AM +0100, Jason A. Donenfeld wrote:
> 
> So far implementing (3) is failing miserably. Is there anything wrong
> with my general idea that might make this a priori impossible? For
> example, will udp_tunnel_xmit_skb not accept super-packets? Or, am I
> just not making use of whatever nice convenience functions are
> available for constructing super-packets, and thus am doing something
> wrong?

I don't see anything fundamentally wrong with your idea.  After
all what you're describing is the basis of GSO, i.e., letting
data stay in the form of super-packets for as long as we can.

Of course there's going to be a lot of niggly bits that you'll
have to sort out to get it to work.

Good luck,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: NETIF_F_GSO_SOFTWARE vs NETIF_F_GSO

[Jason A. Donenfeld]

On Fri, Nov 6, 2015 at 5:58 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> I don't see anything fundamentally wrong with your idea.  After
> all what you're describing is the basis of GSO, i.e., letting
> data stay in the form of super-packets for as long as we can.
>
> Of course there's going to be a lot of niggly bits that you'll
> have to sort out to get it to work.

Okay, great. In that case, it's the niggly bits I need to wrestle
with. Any nice convenience functions for this? Or pointers to
examples?

Would it be best to (1) allocate a massive linear skb, and then bit by
bit call skb_put (as in my example in the last email)? Or would it be
best to (2) make individual packets, and then string them together in
a FRAGLIST fashion (or is there no driver support for this)? Or would
it be best (3) to some how use the array of 16 frags, and allocate
with the alloc_skb_wth_frags function, and then super cumbersomely try
to iterate over broken-up pages?

I think (3) is going to be super hard if not impossible for my setup.
If there were some way of combining multiple skbs into a super packet,
or if the fraglist was okay to use in this regard, that'd be a
lifesaver. So far my approach with (1) hasn't been working.