Re: xennet: skb rides the rocket: 20 slots

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>,
	ANNIE LI <annie.li@oracle.com>,
	xen-devel <xen-devel@lists.xen.org>
Subject: Re: xennet: skb rides the rocket: 20 slots
Date: Thu, 10 Jan 2013 10:39:31 -0500	[thread overview]
Message-ID: <20130110153931.GB13645@phenom.dumpdata.com> (raw)
In-Reply-To: <1357820816.9456.15.camel@zakaz.uk.xensource.com>

On Thu, Jan 10, 2013 at 12:26:56PM +0000, Ian Campbell wrote:
> On Thu, 2013-01-10 at 11:22 +0000, ANNIE LI wrote:
> > I am thinking to do re-fragment in netfront for these skbs like following,
> > 
> > Create a new skb, copy linear data and frag data from original skb into 
> > this one, and make every frags data size is PAGE_SIZE except for the 
> > last fragment. It is possible that the last fragment length is less than 
> > PAGE_SIZE, then free the original skb. The skb packet is large, and 
> > there will be lots of copys.
> 
> You don't need (or I suspect want) to copy, you can directly add pages
> from the source skb's frags to the destination skb's frags, with
> appropriate refcount frobbing. You can also share a page between two (or
> more) skbs in the case where the boundary between two skbs happens to be
> in the middle of a page.

That is allowed by the networking subsystem? I guess it will allow it as
the refcount keeps the users of the skb sane so that you don't inadvertly
pull the page out of the skb (or for highman pages - unmap it).

> 
> But more importantly than all that you need to do more than just
> refragment, you actually need to resegment i.e. you need to duplicate
> the headers (Ethernet, IP, TCP) at the front of each new skb and adjust
> the (psuedo-)checksums as appropriate (which will depend on whether the
> SKB is GSO, or just has checksum offload or nothing).
> 
> So you need to go from
>   <Ether><IP><TCP><...data... ...data...>
> to
>   <Ether><IP><TCP><...data...> <Ether><IP><TCP><...data...>
> 
> Where the headers are adjusted to cope with this. Some of data might be
> in frags but equally some might be in skb->data.
> 
> I'm guessing that Linux already has code which can do this for you,
> since it has a software fallback for GSO.

Hm, my fear is that global extra copying will mean that we lose the benefit
of having this in the cache - as the payload from user-space will not
land on the same cache-lines anymore with this adjustment.

This will require a bit more thinking on how to solve this particular
problem.