All of lore.kernel.org
 help / color / mirror / Atom feed
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Hans Westgaard Ry <hans.westgaard.ry@oracle.com>
Cc: "David S. Miller" <davem@davemloft.net>,
	Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>,
	James Morris <jmorris@namei.org>,
	Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
	Patrick McHardy <kaber@trash.net>,
	Alexei Starovoitov <ast@plumgrid.com>,
	Jiri Pirko <jiri@mellanox.com>,
	Eric Dumazet <edumazet@google.com>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Nicolas Dichtel <nicolas.dichtel@6wind.com>,
	"\"\"Eric W. Biederman\"\"" <ebiederm@xmission.com>,
	Salam Noureddine <noureddine@arista.com>,
	Jarod Wilson <jarod@redhat.com>,
	Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>,
	Julian Anastasov <ja@ssi.bg>, Ying Xue <ying.xue@windriver.com>,
	Craig Gallek <kraig@google.com>,
	Mel Gorman <mgorman@techsingularity.net>,
	"\"\"hannes@stressinduktion.org\"\"" <hannes@stressinduktion.org>,
	Edward Jee <edjee@google.com>,
	Julia Lawall <julia.lawall@lip6.fr>,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	Haakon Bugge <haakon.bugge@oracle.com>,
	Knut Omang <knut.omang@oracle.com>,
	Wei Lin Guay <wei.lin.guay@oracle.com>,
	Santosh Shilimkar <santosh.shilimkar@oracle.com>,
	Yuval Shaia <yuval.shaia@oracle.com>
Subject: Re: [PATCH] net: add per device sg_max_frags for skb
Date: Wed, 06 Jan 2016 06:05:15 -0800	[thread overview]
Message-ID: <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> (raw)
In-Reply-To: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com>

On Wed, 2016-01-06 at 14:16 +0100, Hans Westgaard Ry wrote:
> Devices may have limits on the number of fragments in an skb they
> support. Current codebase uses a constant as maximum for number of
> fragments (MAX_SKB_FRAGS) one skb can hold and use.
> 
> When enabling scatter/gather and running traffic with many small
> messages the codebase uses the maximum number of fragments and thereby
> violates the max for certain devices.
> 
> An example of such a violation is when running IPoIB on a HCA
> supporting 16 SGE on an architecture with 4K pagesize. The
> MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another
> segment we end up with send_requests with 18 SGE resulting in
> kernel-panic.
> 
> The patch allows the device to limit the maximum number fragments used
> in one skb.
> 
> The functionality corresponds to gso_max_size/gso_max_segs for gso.

Unfortunately this is not the right place to fix this issue.

Think about forwarding workloads, where the SKB is cooked by GRO engine.

Anyway, local TCP stack uses 32KB page fragments, so typical skb has no
more than 3 frags.

Look at ndo_features_check(), where the problematic device driver can
add its logic.




WARNING: multiple messages have this Message-ID (diff)
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Hans Westgaard Ry <hans.westgaard.ry@oracle.com>
Cc: "David S. Miller" <davem@davemloft.net>,
	Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>,
	James Morris <jmorris@namei.org>,
	Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
	Patrick McHardy <kaber@trash.net>,
	Alexei Starovoitov <ast@plumgrid.com>,
	Jiri Pirko <jiri@mellanox.com>,
	Eric Dumazet <edumazet@google.com>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Nicolas Dichtel <nicolas.dichtel@6wind.com>,
	"\"\"Eric W. Biederman\"\"" <ebiederm@xmission.com>,
	Salam Noureddine <noureddine@arista.com>,
	Jarod Wilson <jarod@redhat.com>,
	Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>,
	Julian Anastasov <ja@ssi.bg>, Ying Xue <ying.xue@windriver.com>,
	Craig Gallek <kraig@google.com>,
	Mel Gorman <mgorman@techsingularity.net>,
	"\"\"hannes@stressinduktion.org\"\"" <hannes@stressinduktion.org>,
	Edward Jee <edjee@google.com>,
	Julia Lawall <julia.lawall@lip6.fr>,
	netdev@vger.kernel.org, linux-kerne
Subject: Re: [PATCH] net: add per device sg_max_frags for skb
Date: Wed, 06 Jan 2016 06:05:15 -0800	[thread overview]
Message-ID: <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> (raw)
In-Reply-To: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com>

On Wed, 2016-01-06 at 14:16 +0100, Hans Westgaard Ry wrote:
> Devices may have limits on the number of fragments in an skb they
> support. Current codebase uses a constant as maximum for number of
> fragments (MAX_SKB_FRAGS) one skb can hold and use.
> 
> When enabling scatter/gather and running traffic with many small
> messages the codebase uses the maximum number of fragments and thereby
> violates the max for certain devices.
> 
> An example of such a violation is when running IPoIB on a HCA
> supporting 16 SGE on an architecture with 4K pagesize. The
> MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another
> segment we end up with send_requests with 18 SGE resulting in
> kernel-panic.
> 
> The patch allows the device to limit the maximum number fragments used
> in one skb.
> 
> The functionality corresponds to gso_max_size/gso_max_segs for gso.

Unfortunately this is not the right place to fix this issue.

Think about forwarding workloads, where the SKB is cooked by GRO engine.

Anyway, local TCP stack uses 32KB page fragments, so typical skb has no
more than 3 frags.

Look at ndo_features_check(), where the problematic device driver can
add its logic.

  parent reply	other threads:[~2016-01-06 14:05 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-01-06 13:16 [PATCH] net: add per device sg_max_frags for skb Hans Westgaard Ry
2016-01-06 13:16 ` Hans Westgaard Ry
2016-01-06 13:59 ` David Laight
2016-01-06 13:59   ` David Laight
2016-01-08  9:55   ` Hans Westgaard Ry
2016-01-08  9:55     ` Hans Westgaard Ry
2016-01-08 10:33     ` David Laight
2016-01-08 10:33       ` David Laight
2016-01-08 11:47     ` Hannes Frederic Sowa
2016-01-08 11:47       ` Hannes Frederic Sowa
2016-01-13 13:57       ` Hans Westgaard Ry
2016-01-13 13:57         ` Hans Westgaard Ry
2016-01-13 14:19         ` Eric Dumazet
2016-01-13 14:19           ` Eric Dumazet
2016-01-13 14:20           ` Eric Dumazet
2016-01-13 14:20             ` Eric Dumazet
2016-01-13 15:07           ` Hannes Frederic Sowa
2016-01-13 15:07             ` Hannes Frederic Sowa
2016-01-13 15:38           ` David Miller
2016-01-13 15:44             ` Eric Dumazet
2016-01-13 15:44               ` Eric Dumazet
2016-01-13 21:07         ` Eric W. Biederman
2016-01-13 21:07           ` Eric W. Biederman
2016-01-27 13:20     ` [PATCH v2] net:Add sysctl_tcp_sg_max_skb_frags Hans Westgaard Ry
2016-01-27 15:15       ` Eric Dumazet
2016-01-27 18:12         ` Hannes Frederic Sowa
2016-02-01 13:12           ` Hans Westgaard Ry
2016-01-27 20:13       ` David Miller
2016-02-03  8:26     ` [PATCH v3] net:Add sysctl_max_skb_frags Hans Westgaard Ry
2016-02-03  8:26       ` Hans Westgaard Ry
2016-02-03 11:25       ` Herbert Xu
2016-02-03 11:36         ` Hannes Frederic Sowa
2016-02-03 12:20           ` Herbert Xu
2016-02-03 14:03             ` Hannes Frederic Sowa
2016-02-03 14:30             ` Eric Dumazet
2016-02-03 14:30               ` Eric Dumazet
2016-02-03 17:36             ` David Laight
2016-02-03 15:58       ` Alexander Duyck
2016-02-03 16:07         ` Eric Dumazet
2016-02-03 16:07           ` Eric Dumazet
2016-02-03 17:43           ` Alexander Duyck
2016-02-03 17:43             ` Alexander Duyck
2016-02-03 17:54             ` Eric Dumazet
2016-02-03 17:54               ` Eric Dumazet
2016-02-03 18:24               ` Alexander Duyck
2016-02-03 18:24                 ` Alexander Duyck
2016-02-03 19:23                 ` Eric Dumazet
2016-02-03 19:23                   ` Eric Dumazet
2016-02-03 21:03                   ` Alexander Duyck
2016-02-09  9:30       ` David Miller
2016-01-06 14:05 ` Eric Dumazet [this message]
2016-01-06 14:05   ` [PATCH] net: add per device sg_max_frags for skb Eric Dumazet
2016-01-08 10:01   ` Hans Westgaard Ry
2016-01-08 10:01     ` Hans Westgaard Ry

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com \
    --to=eric.dumazet@gmail.com \
    --cc=ast@plumgrid.com \
    --cc=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=ebiederm@xmission.com \
    --cc=edjee@google.com \
    --cc=edumazet@google.com \
    --cc=haakon.bugge@oracle.com \
    --cc=hannes@stressinduktion.org \
    --cc=hans.westgaard.ry@oracle.com \
    --cc=ja@ssi.bg \
    --cc=jarod@redhat.com \
    --cc=jiri@mellanox.com \
    --cc=jmorris@namei.org \
    --cc=julia.lawall@lip6.fr \
    --cc=kaber@trash.net \
    --cc=knut.omang@oracle.com \
    --cc=kraig@google.com \
    --cc=kuznet@ms2.inr.ac.ru \
    --cc=linux-kernel@vger.kernel.org \
    --cc=makita.toshiaki@lab.ntt.co.jp \
    --cc=mgorman@techsingularity.net \
    --cc=netdev@vger.kernel.org \
    --cc=nicolas.dichtel@6wind.com \
    --cc=noureddine@arista.com \
    --cc=santosh.shilimkar@oracle.com \
    --cc=wei.lin.guay@oracle.com \
    --cc=ying.xue@windriver.com \
    --cc=yoshfuji@linux-ipv6.org \
    --cc=yuval.shaia@oracle.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.