From mboxrd@z Thu Jan  1 00:00:00 1970
From: Ian Campbell <Ian.Campbell@citrix.com>
Subject: Re: xennet: skb rides the rocket: 20 slots
Date: Thu, 10 Jan 2013 16:25:14 +0000
Message-ID: <1357835114.9456.66.camel@zakaz.uk.xensource.com>
References: <72958707.20130104172854@eikelenboom.it>
	<1357556115.7989.13.camel@zakaz.uk.xensource.com>
	<50EB8091.90705@oracle.com> <323202711.20130108215503@eikelenboom.it>
	<50ED1800.1080208@oracle.com>
	<20130109150850.GI18395@phenom.dumpdata.com>
	<50EEA46E.7000604@oracle.com>
	<1357820816.9456.15.camel@zakaz.uk.xensource.com>
	<20130110153931.GB13645@phenom.dumpdata.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <20130110153931.GB13645@phenom.dumpdata.com>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>, ANNIE LI <annie.li@oracle.com>, xen-devel <xen-devel@lists.xen.org>
List-Id: xen-devel@lists.xenproject.org

On Thu, 2013-01-10 at 15:39 +0000, Konrad Rzeszutek Wilk wrote:
> On Thu, Jan 10, 2013 at 12:26:56PM +0000, Ian Campbell wrote:
> > On Thu, 2013-01-10 at 11:22 +0000, ANNIE LI wrote:
> > > I am thinking to do re-fragment in netfront for these skbs like following,
> > > 
> > > Create a new skb, copy linear data and frag data from original skb into 
> > > this one, and make every frags data size is PAGE_SIZE except for the 
> > > last fragment. It is possible that the last fragment length is less than 
> > > PAGE_SIZE, then free the original skb. The skb packet is large, and 
> > > there will be lots of copys.
> > 
> > You don't need (or I suspect want) to copy, you can directly add pages
> > from the source skb's frags to the destination skb's frags, with
> > appropriate refcount frobbing. You can also share a page between two (or
> > more) skbs in the case where the boundary between two skbs happens to be
> > in the middle of a page.
> 
> That is allowed by the networking subsystem?

Yes, happens all the time (well not quite, but it happens)

>  I guess it will allow it as
> the refcount keeps the users of the skb sane so that you don't inadvertly
> pull the page out of the skb (or for highman pages - unmap it).

Correct.

> 
> > 
> > But more importantly than all that you need to do more than just
> > refragment, you actually need to resegment i.e. you need to duplicate
> > the headers (Ethernet, IP, TCP) at the front of each new skb and adjust
> > the (psuedo-)checksums as appropriate (which will depend on whether the
> > SKB is GSO, or just has checksum offload or nothing).
> > 
> > So you need to go from
> >   <Ether><IP><TCP><...data... ...data...>
> > to
> >   <Ether><IP><TCP><...data...> <Ether><IP><TCP><...data...>
> > 
> > Where the headers are adjusted to cope with this. Some of data might be
> > in frags but equally some might be in skb->data.
> > 
> > I'm guessing that Linux already has code which can do this for you,
> > since it has a software fallback for GSO.
> 
> Hm, my fear is that global extra copying will mean that we lose the benefit
> of having this in the cache - as the payload from user-space will not
> land on the same cache-lines anymore with this adjustment.

Remember that this only happens at all in extreme or unusual cases. The
common case should be that an skb fits into the negotiated number of
slots.

Also there isn't that much copying. You need to copy the heads from
skb->data to new_skb->data but the frags you can just reference
directly.

> This will require a bit more thinking on how to solve this particular
> problem.